# GitLab Language Server - TypeScript (Experiment)

The GitLab Language Server provides a common interface for other IDE extensions
to build out GitLab functionality. The language server supports:

- GitLab Duo Code Suggestions
- GitLab Duo Chat (coming soon)

For bugs and feature requests, open a
[new issue](https://gitlab.com/gitlab-org/editor-extensions/gitlab-lsp/-/issues/new).

[[_TOC_]]

## Introduction

LSP (Language Server Protocol) is a technology that provides an abstraction layer
between tools that provide analysis results (language servers) and the "consumer"
IDEs (language clients). It provides a generic interface to provide analysis results
in LSP-enabled IDEs. Implement the analysis one time, and all IDEs then benefit.

This project is an LSP-based language-server that serves
[GitLab AI Code Suggestions](https://docs.gitlab.com/ee/user/project/repository/code_suggestions/index.html)
to LSP-enabled IDEs.

The server supports [LSP protocol version `3.17`](https://microsoft.github.io/language-server-protocol/specifications/lsp/3.17/specification/).

This language server leverages the
[`textDocument/completion`](https://microsoft.github.io/language-server-protocol/specifications/lsp/3.17/specification/#textDocument_completion)
LSP method to serve code suggestions. For most editors, this hooks into the IntelliSense
feature, which you can invoke with the <kbd>Ctrl</kbd> + <kbd>space</kbd> <kbd>Ctrl</kbd> + <kbd>N</kbd> shortcuts.

## Run the LSP

The LSP client should be responsible for starting the server process and setting up
the connection. The LSP server supports multiple communication methods, and you
must provide one of them at startup:

- **IPC** - Only for communication with the Node process. The server should be a
  Node app. The server is not published as an NPM package, but you can
  use the `out/node/main.js` module if you've compiled the project locally for testing purposes.

  ```shell
  node 'out/node/main.js' --node-ipc
  ```

- **TCP Socket** - Provide a socket port when starting the server process. The client
  should be waiting on that port for the server to connect.

  ```shell
  /opt/gitlab-lsp --socket=6789
  ```

- **STDIO**

  ```shell
  /opt/gitlab-lsp --stdio
  ```

- **Named Pipes** - The Language Server listens as a named pipe server for the client to connect.

  ```shell
  /opt/gitlab-lsp --pipe=pipeName
  ```

[Check `main.ts`](https://github.com/microsoft/vscode-languageserver-node/blob/main/client/src/node/main.ts#L339)
to see how the `vscode-languageclient` Node JS library handles server startup.

## Use open tabs as context

For better results from GitLab Duo Code Suggestions, ensure that Open Tabs Context is enabled in your IDE settings.
This feature uses the contents of the files currently open in your IDE, to get more
accurate and relevant results from Code Suggestions. Like prompt engineering, these files
give GitLab Duo more information about the standards and practices in your code project.

To get the most benefit from using your open tabs as context, open the files relevant to the code
you want to create, including configuration files. When you start work in a new file,
Code Suggestions offers you suggestions in the new file.

Prerequisites:

- Requires GitLab 17.1 or later. Earlier GitLab versions that support Code Suggestions
  cannot weight the content of open tabs more heavily than other files in your project.
- Requires version 4.14.2 or later of the GitLab Workflow extension for Visual Studio Code.
- GitLab Duo Code Suggestions must be enabled for your project, and
  [configured in the GitLab Workflow plugin](https://gitlab.com/gitlab-org/gitlab-vscode-extension#set-up-code-suggestions).
- Requires a [supported code language](#advanced-context-supported-languages).

1. Download and install a supported version of the GitLab extension from the
   [Visual Studio Marketplace](https://marketplace.visualstudio.com/items?itemName=GitLab.gitlab-workflow).
   For more about configuring the extension, see
   [its setup instructions](https://gitlab.com/gitlab-org/gitlab-vscode-extension#setup).
1. Open the files you want to provide for context. Advanced Context uses the most recently
   opened or changed files for context. If you don't want a file sent as additional context, close it.
1. To fine-tune your Code Generation results, add code comments to your file that explain
   what you want to build. Code Generation treats your code comments like chat. Your code comments
   update the `user_instruction`, and then improve the next results you receive.

As you work, GitLab Duo provides code suggestions that use your other open files
(within [truncation limits](https://docs.gitlab.com/ee/user/project/repository/code_suggestions/#truncation-of-file-content))
as extra context.

To learn about the code that builds the prompt, see these files:

- **Code Generation**:
  [`ee/lib/api/code_suggestions.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/lib/api/code_suggestions.rb#L76)
  in the `gitlab` repository
- **Code Completion**:
  [`ai_gateway/code_suggestions/processing/completions.py`](https://gitlab.com/gitlab-org/modelops/applied-ml/code-suggestions/ai-assist/-/blob/fcb3f485a8f047a86a8166aad81f93b6d82106a7/ai_gateway/code_suggestions/processing/completions.py#L273)
  in the `modelops` repository

We'd love your feedback about the Advanced Context feature in
[issue 258](https://gitlab.com/gitlab-org/editor-extensions/gitlab-lsp/-/issues/258).

### Advanced Context supported languages

The Advanced Context feature supports these languages:

- Code Completion: all configured languages.
- Code Generation: Go, Java, JavaScript, Kotlin, Python, Ruby, Rust, TypeScript (`.ts` and `.tsx` files), Vue, and YAML.

## Install the language server client

To install the language server binary locally:

1. Download the language server binary from the
   [Package Registry page](https://gitlab.com/gitlab-org/editor-extensions/gitlab-lsp/-/packages).
   For every release you can download the binary that matches your OS and architecture.
   In the list of assets linked in the release, binaries end with the string `<os>-<arch>`.
1. Place the binary anywhere on your file system. The configuration in this documentation
   assumes you've placed the binary under `/opt/gitlab-lsp`.
1. The language server requires an access token to authenticate with the GitLab API.
   Create a personal access token (PAT) with the `api` scope, or
   OAuth token with the same scope. Provide this token to the server sending the
   [`didChangeConfiguration` notification](https://microsoft.github.io/language-server-protocol/specifications/lsp/3.17/specification/#workspace_didChangeConfigurationworkspace/didChangeConfiguration)
   from the client to the server after the server is started.
   [See the details here](docs/supported_messages.md#didchangeconfiguration).
1. In your terminal, make the binary executable with this command:

   ```shell
   chmod +x opt/gitlab-lsp
   ```

1. The client implementation depends on your text-editor or IDE. See the next section.

### Allow the binary to run on macOS

MacOS users might encounter a message in the terminal, saying the binary was killed
as soon as you started it. This happens because of a security feature called Gatekeeper.
To allow the binary to run, clear its extended attributes and ad-hoc sign it,
which establishes trust with Gatekeeper.

Run these commands in your terminal, and enter your password if prompted:

```shell
sudo xattr -c opt/gitlab-lsp
codesign --force -s - opt/gitlab-lsp
```

## LSP client configuration

- [VS Code](https://gitlab.com/shekharpatnaik/gitlab-lsp-test-extension/-/blob/main/client/src/extension.ts?ref_type=heads#L36-87)

_This section is still in progress..._

## Node version

We use the Node version defined in the
[`Microsoft/vscode` project](https://github.com/microsoft/vscode/blob/main/.nvmrc).
We must make sure to keep the *major* version the same as
what the `pkg-fetch` module supports.

The `pkg-fetch` module provides the node version used by `pkg` when packaging the
language server. Each version of `pkg-fetch` has a static set of node builds.
You can look up the versions on the
[`pkg-fetch` releases page](https://github.com/vercel/pkg-fetch/releases) of the GitHub project.

Prerequisites:

- Make sure the major version is the same for the `pkg` and the development versions.

To upgrade the version of Node in use:

1. Update `pkg` version, if there is a new one:
   1. Upgrade the `pkg-fetch` module by editing the `package.json` file's `overrides` section manually.
      - Note: To get the updated package, make sure to `rm -rf node_modules` and run `npm install`.
   1. Check the [`pkg-fetch` releases page](https://github.com/vercel/pkg-fetch/releases)
      for the Node version. Verify `macos-arm64` builds exist for the selected version.
   1. Update `.gitlab-ci.yml` (`test-pkg-node`) with the new version number.
1. Update the development/VS Code version:
   1. Update `.tool-versions` file with the new version number.
   1. Update `.gitlab-ci.yml` (`default`) with the new version number.
   1. Update `src/tests/int/node_version.test.ts` with the correct version number.
   1. Update `package.json` `scripts` section where we reference the target Node version, like this: `--target=node18.17`.
1. Add a changelog entry.

## Development setup

To debug or add new features to the language server:

1. Clone this repository.
1. Run `npm install`.
1. Compile the server code with `npm run compile` or `npm run watch` to
   compile when the changes are made.

### Watch mode

Watch mode rebuilds the project automatically with every file change. Optionally,
it can also update `gitlab-lsp` dependency in VS Code. For details, see the
[watch mode script](scripts/watcher/watch.ts).

This example uses VS Code. Run this command:

```shell
npm run watch -- --editor=vscode
```

Then, in VS Code, you can restart your extension development host, and it
receives the latest server changes.

#### Script parameters

These parameters are optional:

- `-e, --editor <type>` - Specify editor type (e.g., vscode). If set to 'vscode', the script will copy the build files to automatically update `gitlab-lsp` in your VS Code Extension.
  - **note**: You do not need to restart the extension development host if you wish to restart the server with your changes. Simply run the "Restart Language Server" command in the command palette.
- `-vp, --vscode-path <path>` - Path to VSCode extension
- `-p, --packages <packages...>` - Specify packages to watch. If not provided, all packages will be built.

Supported editors:

- vscode

Example usage:

```shell
npm run watch -- --editor vscode --vscode-path ../my-vscode-extension --packages package1 package2
```

This will watch for changes, build only the specified packages, and copy the results to the specified VS Code extension path.

### Start the server with test VS Code extension

You should test the LSP server together with the test LSP client. See the
[test LSP client (VS Code extension) documentation](https://gitlab.com/shekharpatnaik/gitlab-lsp-test-extension#running-the-client)
for how to configure the test client to run and start the server.

### Start with another client, or without the client

To start the server separately, either:

- Run `npm run debug`.
- Run this command:

  ```shell
  node --inspect=6010 ./out/node/main.js --socket=6789
  ```

Your client should establish the socket connection, and wait on port `6789` for the server to connect.

### Debug the server

To debug the Language Server, you can:

- [Start the Language Server](#start-the-language-server).
- [Connect to the Language Server in the VS Code extension](#connect-to-ls-in-the-vs-code-extension).
- [Connect to the Language Server using Chrome Developer Tools](#connect-to-ls-with-chrome-developer-tools).

#### Start the Language Server

1. Open the project in VS Code.
1. Run the **Start server** launch task.

The launch task starts a Language Server that listens on port `6789` for a client connection,
and connects VS Code debugger to it.

#### Connect to LS in the VS Code extension

Prerequisites:

- If connecting with the VS Code extension, you must have the `code` command configured. See
  [Launching VS Code from the command line](https://code.visualstudio.com/docs/setup/mac#_launching-from-the-command-line) for more information.

1. [Set up the extension project](https://gitlab.com/gitlab-org/gitlab-vscode-extension/blob/main/docs/developer/language-server.md)
   for LS development.
1. [Start the VS Code extension](https://gitlab.com/gitlab-org/gitlab-vscode-extension/-/blob/main/CONTRIBUTING.md?ref_type=heads#step-4-running-the-extension-in-desktop-vs-code) in development mode.
1. Use the command line to open this project in VS Code, replacing the path with
   the path of your VS Code Extension project:

   ```shell
   GITLAB_WORKFLOW_PATH=/Users/tomas/workspace/gitlab-vscode-extension code .
   ```

   This command is important. It maps the bundled sources in the extension with the source code in this project.

1. Run the **Attach to VS Code Extension** launch task.

#### Connect to LS with Chrome Developer Tools

Prerequisites:

- Your server must be a Node module, not an executable.

1. Either:
   - [Start the VS Code extension](https://gitlab.com/gitlab-org/gitlab-vscode-extension/-/blob/main/CONTRIBUTING.md?ref_type=heads#step-4-running-the-extension-in-desktop-vs-code) in development mode, or
   - [Start the server](#start-with-another-client-or-without-the-client).
1. Open [`chrome://inspect`](chrome://inspect) in a Chrome browser. For more information and
   debugging tools, see [Inspector Clients](https://nodejs.org/en/docs/guides/debugging-getting-started#inspector-clients)
   in the Node.js documentation.
1. Select **Configure**, and ensure your target host and port are listed. Use `localhost:6010`.
1. Select **Open dedicated DevTools for Node**.
1. Either add the `debugger` statement, or add `console.log` in your server code.
1. Recompile after you change the code.

When you start the Language Server, the inspector pauses on the breakpoints.

### Test changes in downstream project pipelines

You can install the artifact created by the `build_package_for_integration` job
in merge request pipelines in downstream projects. This allows you to run CI/CD
for the downstream project without waiting for a new Language Server version to
be published.

For instance, in the VS Code extension, you can run:

```shell
npm install <URL>
```

where `URL` is the URL reported in the job log. Then, you can commit the
`package*.json` changes and push them to a merge request.

### Lint documentation

This project follows the documentation guidelines of GitLab and uses `markdownlint`
and `vale` to improve the content added and make sure it adheres to our guidelines.
You can find information on
[installing the linters](https://docs.gitlab.com/ee/development/documentation/testing.html#install-linters) or
[integrating with your editor](https://docs.gitlab.com/ee/development/documentation/testing.html#configure-editors)
in the GitLab documentation.

## Bundling

We need to create a self-contained bundled JavaScript file that is started by
VS Code. We use `esbuild` to bundle all the dependencies into one file located in
`out/node/main-bundle.js`.

## Packaging

Run:

```shell
npm run package
```

Your binary is available in the `bin` folder of the project.

## Releases

Releases consist of deploying an NPM package and a corresponding generic package to
the [package registry](https://gitlab.com/gitlab-org/editor-extensions/gitlab-lsp/-/packages).

To release a new version of the packages:

1. Create a new branch `git checkout -b 2023-11-06-release`
1. Run `npm version major/minor/patch`. Use `major`, `minor`, or `patch` as defined by semantic versioning.
   - This command creates a new version, updates the changelog, and tags the commit.
1. Push the branch and the tag. For example, if you created `v3.4.5`, run this command:

   ```shell
   git push && git push origin v3.4.5
   ```

1. Create a merge request from the `2023-11-06-release` branch:
   - Use the version number in the merge request name, like this: `Release LS version 3.4.5`.
   - Make sure the merge request **is NOT** set to squash, because squashing changes the release commit SHA.
1. No review is needed, as no code has changed.
1. Merge the `Release LS version 3.4.5` merge request.

### Communicate updates to downstream extensions

Communicate updates to the downstream projects through integration epics. An epic
contains upgrade issues for each internal project that consumes the Language Server.
[Example](https://gitlab.com/groups/gitlab-org/editor-extensions/-/epics/2).

Once per week, if any new releases after the last integration epic would benefit
the downstream projects, create a new integration epic.

## Troubleshooting

## Advanced Context feature and memory

The Advanced Context feature stores file information in memory. If you
encounter memory errors, use fewer files to generate context, and close
any you don't need.