Merge pull request #280 from hausdorff/registries

Design: propose improvements to the "fresh clone" story

design/README.md

+# Design Documents
+
+This directory contains a set of design documents that describe the design and
+implementation of the ksonnet toolchain. This includes not only the ksonnet CLI
+tool, but all the surrounding tooling as well, such as the language server, the
+core libraries, _etc_.
+
+As significant feature work will be accompanied by design documents,
+this directory is a living representation of the ksonnet toolchain.
+
+## CLI
+
+* [`ks registry update` and `ks registry delete`](https://github.com/ksonnet/ksonnet/tree/master/design/proposals/ks-registry-update-delete.md)
+* [Improve the "fresh clone" story](https://github.com/ksonnet/ksonnet/tree/master/design/proposals/fresh-clone.md)

design/proposals/fresh-clone.md

+# Improve the "fresh-clone" story
+
+Status: Pending
+
+Version: Alpha
+
+## Abstract
+
+This document proposes to extend the dependency, registry, and environment
+management code to provide a cleaner "fresh-clone" story.
+
+Essentially, a user should only need to run `git clone` and `ks pkg install` to
+get started developing, and it should not be necessary to check bulky metadata
+files into a version control system to have a good clone-to-development story.
+In this sense `ks pkg install` would take on a role similar to `npm install`,
+which is used to obtain all dependencies of a node.js project.
+
+## Motivation
+
+The `ks` CLI tool is largely awkward to use with source control systems like
+`git`. There are several reasons for this, but likely the biggest is that
+ksonnet keeps around a lot of metadata that can (and should) be generated
+automatically. Because it's not, users are forced to keep this in version
+control, which is non-ideal.
+
+Two good examples of this problem are:
+
+1. source code for packages the application depends on, and
+1. metadata associated with ksonnet _environments_ (_e.g._, information about
+   the cluster, the `k8s.libsonnet` file that is generated based on the OpenAPI
+   specification associated with that cluster, and so on).
+
+(2) in particular is costly; each `k8s.libsonnet` is about ~1.7MB.
+
+Since it is possible generate (or acquire) all of these things based on the
+information in the `app.yaml` file, this commit proposes to introduce a `ks
+install` command. This command follows the example set by NPM, a project with a
+similar problem; NPM has a `package.json` file, and `npm install` is used to
+(_e.g._) pull down dependencies from the NPM registry.
+
+The related issue is captured in [#217][217].
+
+## Goal
+
+1. Make it trivial to clone and get started developing a ksonnet application (it
+   should be two commands: `git clone` followed by `ks pkg install`).
+1. Reduce the memory footprint of checking a ksonnet application into a source
+   control system like `git` by autogenerating bulky metadata files.
+1. Auto-generate a `.gitignore` as a result of `ks init`. (See issue [#55][55])
+1. Don't block development because the user doesn't have access to an
+   environment. For example, `prod` environments are usually locked down; a user
+   should not be prevented from developing a ksonnet application simply because
+   they don't have access to this environment. (They will obviously not be able
+   to push to that environment, however.)
+1. When `ks pkg install` runs, it should present a clear, easy-to-consume
+   picture of what it's going to do (or has done). For example, it should say
+   "installed these dependencies", _etc_.
+
+## Proposal
+
+This proposal consists mainly of two main bodies of work:
+
+1. changes to the semantics and implementation of `ks pkg install` (which will
+   attempt to get all dependencies if they are not already present)
+1. changes to the way environments are initialized (_i.e._, we can't expect
+   users to have access to all environments, so the `ks` CLI needs to be able to
+   tolerate this).
+
+### `ks pkg install`
+
+We propose the form of the command should change from `ks pkg install
+<pkg-name>` to `ks pkg install [pkg-name]` (_i.e._, we propose `pkg-name` should
+be optional).
+
+When run with no `pkg-name`, `ks pkg install` should cause all dependencies to
+be installed if not present. Specifically:
+
+1. We traverse the list of registries in `app.yaml` and attempt to retrieve and
+   cache each of them locally.
+1. We traverse the list of dependencies in `app.yaml` and attempt to install
+   them one by one.
+
+Other changes related to partially-initialized ksonnet applications are not
+necessary. For example, if a user runs `git clone` and forgets to run `ks pkg
+install`, we should allow the Jsonnet compiler to not find the dependency, and
+error out. If this becomes an expressed problem from users, we can re-evaluate.
+
+This notably does _not_ solve the problem of generating the environment
+metadata. This is handled in the next section.
+
+### Lazily-instantiated environments
+
+A number of commands depend on a fully-initialized environment. Since we cannot
+expect users to have access to all environments (_e.g._, most `prod`
+environments are locked down), the `ks` CLI tool needs to tolerate this case
+when the locked-down environments aren't needed, and it needs to present useful
+errors in the case they are.
+
+To resolve this problem, we propose that environments should be _instantiated
+lazily_ (by which we mean _when needed_ rather than ahead of time).
+Specifically, we propose that commands that interact with the environment
+(_e.g._, `ks diff`, `ks apply`, and so on) have the following workflow:
+
+1. Check whether the environment is initialized.
+1. If it is, try to run the command.
+1. If it's not, attempt to initialize the environment from the existing
+   metadata.
+
+## User Experience
+
+### Backwards compatibility
+
+This change will be backwards-compatible.
+
+## Alternatives considered
+
+The only alternative is to do what we are doing now, which is to force users to
+check the metadata files into source control.
+
+[55]: https://github.com/ksonnet/ksonnet/issues/55
+[217]: https://github.com/ksonnet/ksonnet/issues/217

design/proposals/ks-registry-update-delete.md

+# `ks registry update` and `ks registry delete`
+
+Status: Pending
+
+Version: Alpha
+
+## Abstract
+
+This proposal describes `ks registry update` and `ks registry delete`, commands
+that will allow users to update and delete the local cache of their registry.
+This is intended to provide a clear path for users to obtain new libraries that
+have been added to the source registry, as well as new versions of old libraries
+in a registry.
+
+## Motivation
+
+The `ks` client tool currently allows user to manage _registries_—collections of
+_packages_ that can be installed and used in the local ksonnet application.
+
+Managing the local cache of a registry when the two diverge is a major painpoint
+in the `ks` client tool. In particular, `ks` currently does not have the ability
+either to update the registry, or to delete a registry, so when a new package is
+added to a remote registry, in order to obtain it, they must:
+
+1. figure out how registries work (_i.e._, via the registry specification in a
+   YAML file), and
+2. manually update their registry spec files (and possibly script it, as [this
+   user has done])
+
+Since users are now creating their own custom registries, we can't afford for
+this part of the workflow to be painful.
+
+The related issue is captured in [#237][237].
+
+## Goal
+
+* Make it trivial to additively refresh registries (_i.e._, most updates should
+  be one command).
+* Make it trivial to delete a registry (_i.e._, if we don't depend on anything
+  from a registry, it should take one command).
+* The `registry update` and `registry delete` commands should output a
+  trivially-consumable summary of changes that are being incorporated. (_e.g._,
+  "adding these packages", "removing these", "adding releases for these
+  packages")
+* Should alert the user if a `registry update` or `registry delete` is about to
+  make the application un-buildable. For example, if `registry delete`'ing the
+  registry would delete a version of a dependency you actually use, we should
+  warn the user before proceeding (perhaps requiring a `--force` flag).
+
+## Non Goals
+
+* Any sort of package refresh. Something like `ks pkg upgrade` is orthogonal to
+  the current expressed pain of users, and is complex enough to greatly slow the
+  speed of delivering the cause of this pain (namely, `ks registry update`).
+* Transitive closure resolution. Our package structure is very simple, so we
+  don't need to traverse dependencies of dependencies (yet!).
+
+## Proposal
+
+We propose to implement the following commands:
+
+```
+ks registry update <registry-name> [flags]
+
+Attempt to update our local cache of a package registry. This might include
+adding or removing packages from the cached registry, as well as adding or
+removing specific versions of a package. These packages can be installed into
+the current application with `ks pkg`.
+
+This command additionally provides sanity checks, and will fail in cases where
+updating would make the application un-buildable. For example, if 'registry
+update' would remove a package (or a version of a package) that the application
+depends on, the command will fail. This can be overridden using the `--force`
+command.
+
+Usage:
+  ks registry update <registry-name> [flags]
+
+Flags:
+  -h, --help   help for update
+      --force  Ignore warnings and force the update
+
+Global Flags:
+  -v, --verbose count[=-1]   Increase verbosity. May be given multiple times.
+```
+
+```
+ks registry delete <registry-name> [flags]
+
+Attempt to delete our local cache of a package registry. The application will
+not be able to use packages from this registry after it is deleted.
+
+This command additionally provides sanity checks, and will fail in cases where
+deleting would make the application un-buildable. For example, if 'registry
+delete' would remove a package that the application depends on, the command
+will fail. This can be overridden using the `--force` command.
+
+Usage:
+  ks registry delete <registry-name> [flags]
+
+Flags:
+  -h, --help   help for delete
+      --force  Ignore warnings and force the delete
+
+Global Flags:
+  -v, --verbose count[=-1]   Increase verbosity. May be given multiple times.
+```
+
+## User Experience
+
+### Backwards compatibility
+
+This change will be backwards-compatible.
+
+## Alternatives considered
+
+The only alternative is essentially to allow users to do this manually; this
+user wrote a [script][script] to do it.
+
+[237]: https://github.com/ksonnet/ksonnet/issues/237
+[script]: https://github.com/ksonnet/ksonnet/issues/237#issuecomment-356268108