diff --git a/design/README.md b/design/README.md new file mode 100644 index 0000000000000000000000000000000000000000..028197e94b8c4f01a636ed1a9ca1f721694c2eaf --- /dev/null +++ b/design/README.md @@ -0,0 +1,14 @@ +# Design Documents + +This directory contains a set of design documents that describe the design and +implementation of the ksonnet toolchain. This includes not only the ksonnet CLI +tool, but all the surrounding tooling as well, such as the language server, the +core libraries, _etc_. + +As significant feature work will be accompanied by design documents, +this directory is a living representation of the ksonnet toolchain. + +## CLI + +* [`ks registry update` and `ks registry delete`](https://github.com/ksonnet/ksonnet/tree/master/design/proposals/ks-registry-update-delete.md) +* [Improve the "fresh clone" story](https://github.com/ksonnet/ksonnet/tree/master/design/proposals/fresh-clone.md) diff --git a/design/proposals/fresh-clone.md b/design/proposals/fresh-clone.md new file mode 100644 index 0000000000000000000000000000000000000000..3bfc1e785303127c6aac5c3d6605bb93c5f66b6c --- /dev/null +++ b/design/proposals/fresh-clone.md @@ -0,0 +1,121 @@ +# Improve the "fresh-clone" story + +Status: Pending + +Version: Alpha + +## Abstract + +This document proposes to extend the dependency, registry, and environment +management code to provide a cleaner "fresh-clone" story. + +Essentially, a user should only need to run `git clone` and `ks pkg install` to +get started developing, and it should not be necessary to check bulky metadata +files into a version control system to have a good clone-to-development story. +In this sense `ks pkg install` would take on a role similar to `npm install`, +which is used to obtain all dependencies of a node.js project. + +## Motivation + +The `ks` CLI tool is largely awkward to use with source control systems like +`git`. There are several reasons for this, but likely the biggest is that +ksonnet keeps around a lot of metadata that can (and should) be generated +automatically. Because it's not, users are forced to keep this in version +control, which is non-ideal. + +Two good examples of this problem are: + +1. source code for packages the application depends on, and +1. metadata associated with ksonnet _environments_ (_e.g._, information about + the cluster, the `k8s.libsonnet` file that is generated based on the OpenAPI + specification associated with that cluster, and so on). + +(2) in particular is costly; each `k8s.libsonnet` is about ~1.7MB. + +Since it is possible generate (or acquire) all of these things based on the +information in the `app.yaml` file, this commit proposes to introduce a `ks +install` command. This command follows the example set by NPM, a project with a +similar problem; NPM has a `package.json` file, and `npm install` is used to +(_e.g._) pull down dependencies from the NPM registry. + +The related issue is captured in [#217][217]. + +## Goal + +1. Make it trivial to clone and get started developing a ksonnet application (it + should be two commands: `git clone` followed by `ks pkg install`). +1. Reduce the memory footprint of checking a ksonnet application into a source + control system like `git` by autogenerating bulky metadata files. +1. Auto-generate a `.gitignore` as a result of `ks init`. (See issue [#55][55]) +1. Don't block development because the user doesn't have access to an + environment. For example, `prod` environments are usually locked down; a user + should not be prevented from developing a ksonnet application simply because + they don't have access to this environment. (They will obviously not be able + to push to that environment, however.) +1. When `ks pkg install` runs, it should present a clear, easy-to-consume + picture of what it's going to do (or has done). For example, it should say + "installed these dependencies", _etc_. + +## Proposal + +This proposal consists mainly of two main bodies of work: + +1. changes to the semantics and implementation of `ks pkg install` (which will + attempt to get all dependencies if they are not already present) +1. changes to the way environments are initialized (_i.e._, we can't expect + users to have access to all environments, so the `ks` CLI needs to be able to + tolerate this). + +### `ks pkg install` + +We propose the form of the command should change from `ks pkg install +<pkg-name>` to `ks pkg install [pkg-name]` (_i.e._, we propose `pkg-name` should +be optional). + +When run with no `pkg-name`, `ks pkg install` should cause all dependencies to +be installed if not present. Specifically: + +1. We traverse the list of registries in `app.yaml` and attempt to retrieve and + cache each of them locally. +1. We traverse the list of dependencies in `app.yaml` and attempt to install + them one by one. + +Other changes related to partially-initialized ksonnet applications are not +necessary. For example, if a user runs `git clone` and forgets to run `ks pkg +install`, we should allow the Jsonnet compiler to not find the dependency, and +error out. If this becomes an expressed problem from users, we can re-evaluate. + +This notably does _not_ solve the problem of generating the environment +metadata. This is handled in the next section. + +### Lazily-instantiated environments + +A number of commands depend on a fully-initialized environment. Since we cannot +expect users to have access to all environments (_e.g._, most `prod` +environments are locked down), the `ks` CLI tool needs to tolerate this case +when the locked-down environments aren't needed, and it needs to present useful +errors in the case they are. + +To resolve this problem, we propose that environments should be _instantiated +lazily_ (by which we mean _when needed_ rather than ahead of time). +Specifically, we propose that commands that interact with the environment +(_e.g._, `ks diff`, `ks apply`, and so on) have the following workflow: + +1. Check whether the environment is initialized. +1. If it is, try to run the command. +1. If it's not, attempt to initialize the environment from the existing + metadata. + +## User Experience + +### Backwards compatibility + +This change will be backwards-compatible. + +## Alternatives considered + +The only alternative is to do what we are doing now, which is to force users to +check the metadata files into source control. + +[55]: https://github.com/ksonnet/ksonnet/issues/55 +[217]: https://github.com/ksonnet/ksonnet/issues/217 diff --git a/design/proposals/ks-registry-update-delete.md b/design/proposals/ks-registry-update-delete.md new file mode 100644 index 0000000000000000000000000000000000000000..35d6021aec83d64442087ec4964fec64c70b4dff --- /dev/null +++ b/design/proposals/ks-registry-update-delete.md @@ -0,0 +1,121 @@ +# `ks registry update` and `ks registry delete` + +Status: Pending + +Version: Alpha + +## Abstract + +This proposal describes `ks registry update` and `ks registry delete`, commands +that will allow users to update and delete the local cache of their registry. +This is intended to provide a clear path for users to obtain new libraries that +have been added to the source registry, as well as new versions of old libraries +in a registry. + +## Motivation + +The `ks` client tool currently allows user to manage _registries_—collections of +_packages_ that can be installed and used in the local ksonnet application. + +Managing the local cache of a registry when the two diverge is a major painpoint +in the `ks` client tool. In particular, `ks` currently does not have the ability +either to update the registry, or to delete a registry, so when a new package is +added to a remote registry, in order to obtain it, they must: + +1. figure out how registries work (_i.e._, via the registry specification in a + YAML file), and +2. manually update their registry spec files (and possibly script it, as [this + user has done]) + +Since users are now creating their own custom registries, we can't afford for +this part of the workflow to be painful. + +The related issue is captured in [#237][237]. + +## Goal + +* Make it trivial to additively refresh registries (_i.e._, most updates should + be one command). +* Make it trivial to delete a registry (_i.e._, if we don't depend on anything + from a registry, it should take one command). +* The `registry update` and `registry delete` commands should output a + trivially-consumable summary of changes that are being incorporated. (_e.g._, + "adding these packages", "removing these", "adding releases for these + packages") +* Should alert the user if a `registry update` or `registry delete` is about to + make the application un-buildable. For example, if `registry delete`'ing the + registry would delete a version of a dependency you actually use, we should + warn the user before proceeding (perhaps requiring a `--force` flag). + +## Non Goals + +* Any sort of package refresh. Something like `ks pkg upgrade` is orthogonal to + the current expressed pain of users, and is complex enough to greatly slow the + speed of delivering the cause of this pain (namely, `ks registry update`). +* Transitive closure resolution. Our package structure is very simple, so we + don't need to traverse dependencies of dependencies (yet!). + +## Proposal + +We propose to implement the following commands: + +``` +ks registry update <registry-name> [flags] + +Attempt to update our local cache of a package registry. This might include +adding or removing packages from the cached registry, as well as adding or +removing specific versions of a package. These packages can be installed into +the current application with `ks pkg`. + +This command additionally provides sanity checks, and will fail in cases where +updating would make the application un-buildable. For example, if 'registry +update' would remove a package (or a version of a package) that the application +depends on, the command will fail. This can be overridden using the `--force` +command. + +Usage: + ks registry update <registry-name> [flags] + +Flags: + -h, --help help for update + --force Ignore warnings and force the update + +Global Flags: + -v, --verbose count[=-1] Increase verbosity. May be given multiple times. +``` + +``` +ks registry delete <registry-name> [flags] + +Attempt to delete our local cache of a package registry. The application will +not be able to use packages from this registry after it is deleted. + +This command additionally provides sanity checks, and will fail in cases where +deleting would make the application un-buildable. For example, if 'registry +delete' would remove a package that the application depends on, the command +will fail. This can be overridden using the `--force` command. + +Usage: + ks registry delete <registry-name> [flags] + +Flags: + -h, --help help for delete + --force Ignore warnings and force the delete + +Global Flags: + -v, --verbose count[=-1] Increase verbosity. May be given multiple times. +``` + +## User Experience + +### Backwards compatibility + +This change will be backwards-compatible. + +## Alternatives considered + +The only alternative is essentially to allow users to do this manually; this +user wrote a [script][script] to do it. + +[237]: https://github.com/ksonnet/ksonnet/issues/237 +[script]: https://github.com/ksonnet/ksonnet/issues/237#issuecomment-356268108