# `renovate-graph`, the Renovate package data exporter

`renovate-graph` is a tool, powered by [Renovate](https://docs.renovatebot.com/), that allows exporting the dependency tree of a repository as a JSON blob.

It's named like so because it is a lightweight wrapper on top of the [Renovate](https://docs.renovatebot.com/) dependency update tool, extending Renovate with the ability to collect additional package data against repositories, and exporting the data in a form that can be used for other purposes.

`renovate-graph` leverages the [dozens of supported package ecosystems](https://docs.renovatebot.com/modules/manager/) that Renovate does, as well as [the ability to define your own](https://docs.renovatebot.com/modules/manager/regex/).

However, as `renovate-graph` is purely used to extract package metadata, it ignores the way that Renovate is configured for different workflows, allowing you to extract _all_ package metadata, even if a repository disables it i.e. via [`enabledManagers`](https://docs.renovatebot.com/configuration-options/#enabledmanagers).

## FAQ

### Do I already need to use Renovate?

No, you don't need to be onboarded to Renovate to use `renovate-graph`, and comes with a reasonable set of defaults.

### Do I have to use `renovate-graph` if we're already using Renovate?

If you are using Renovate already, you'll get better data out of `renovate-graph`.

It's recommended you run `renovate-graph` _in addition_ to Renovate, given `renovate-graph` can ignore any disabling of dependency scanning (i.e. via [`enabledManagers`](https://docs.renovatebot.com/configuration-options/#enabledmanagers) or [`packageRules`](https://docs.renovatebot.com/configuration-options/#packagerules)), which allows it to collect more data than may be found by Renovate itself.

Repository configuration will be followed, but disabling of dependency scanning will be ignored.

### Why would I use `renovate-graph` over Renovate's inbuilt data export functionality?

Aha, you know your Renovate configuration well 👀

You're likely referring to the mechanisms described in [GitHub Discussion: create dependency list / tree](https://github.com/renovatebot/renovate/discussions/13150) or [GitHub Discussion: Add support for outputting result of extracted package tree and pending updates](https://github.com/renovatebot/renovate/discussions/24823).

If so, you referring to one of the following:

#### Renovate debug logs

When Renovate is configured to output debug logs i.e. via [the `LOG_LEVEL=debug` environment variable](https://docs.renovatebot.com/config-overview/#logging-variables), you will see log lines such as:

```json

{
  "baseBranch": "main",
  "config": {
    "github-actions": [
      {
        "deps": [
          { ... }
        ],
        "packageFile": ".github/workflows/rebuild.yml"
      }
    ]
  },
  "hostname": "...",
  "level": 20,
  "logContext": "...",
  "msg": "packageFiles with updates",
  "name": "renovate",
  "pid": 1,
  "repository": "jamietanna/jamietanna",
  "time": "...",
  "v": 0
}
```

This can be parsed to receive similar data to what `renovate-graph` exports.

**NOTE**: When using an existing Renovate setup, this is susceptible to missing data due to the way that Renovate is configured, for instance to [require configuration](https://docs.renovatebot.com/self-hosted-configuration/#requireconfig), explicitly opting out a repository or limiting the [`enabledManagers`](https://docs.renovatebot.com/configuration-options/#enabledmanagers), all of which will report different data to that of `renovate-graph`, which ignores any repo-specific configuration. This is suboptimal data collection, but is still _a lot_ better than other tools on the market, and is better than _not_ using Renovate data.

If you are happy with this, then you can [consume these via Dependency Management Data (aka DMD)](https://dmd.tanna.dev/cookbooks/consuming-renovate-debug-logs/).

#### Renovate Reports

When Renovate is configured to perform [Report exports](https://docs.renovatebot.com/self-hosted-configuration/#reporttype), you will receive a file export such as:

```json
{
  "problems": [
  ],
  "repositories": {
    "jamietanna/jamietanna": {
      "branches": [

      ],
      "packageFiles": {
        "github-actions": [
          { ... }
        ],
        "packageFile": ".github/workflows/rebuild.yml"
      }
    },
    "oapi-codegen/oapi-codegen": {
      "branches": [

      ],
      "packageFiles": {
        "gomod": [
          { ... }
        ]
      }
    }
  }
}
```

This can be parsed to receive similar data to what `renovate-graph` exports.

**NOTE**: When using an existing Renovate setup, this is susceptible to missing data due to the way that Renovate is configured, for instance to [require configuration](https://docs.renovatebot.com/self-hosted-configuration/#requireconfig), explicitly opting out a repository or limiting the [`enabledManagers`](https://docs.renovatebot.com/configuration-options/#enabledmanagers), all of which will report different data to that of `renovate-graph`, which ignores any repo-specific configuration. This is suboptimal data collection, but is still _a lot_ better than other tools on the market, and is better than _not_ using Renovate data.

If you are happy with this, then you can [consume these via Dependency Management Data (aka DMD)](https://dmd.tanna.dev/cookbooks/consuming-renovate-report/).

### How can I use the data?

`renovate-graph` produces an unminified, but human readable, JSON blob that can be used by other tools for any other purposes.

However, there are a few tools that are best used with `renovate-graph`, under the [Dependency Management Data (aka DMD)](https://dmd.tanna.dev) project:

- Importing the data via the [`dmd import renovate` command](https://dmd.tanna.dev/commands/dmd_import_renovate/), detailed more under the [Getting Started guide](https://dmd.tanna.dev/cookbooks/getting-started/)
  - This then produces an SQLite database which can be further queried
- [`renovate-to-sbom`](https://dmd.tanna.dev/commands/renovate-to-sbom/), to convert a data dump into a (low fidelity) Software Bill of Materials (SBOM)
- [`renovate-packagedata-diff`](https://dmd.tanna.dev/commands/renovate-packagedata-diff/), to perform a semantic diff between two data exports

You're also very free to do what you want with it, as it's all open data 🚀

### What does the data look like?

You can find a number of example `renovate-graph` data exports in the [dependency-management-data-example project](https://gitlab.com/tanna.dev/dependency-management-data-example/-/tree/main/renovate?ref_type=heads), from a range of public projects.

### Why is it called `renovate-graph` if it doesn't provide a full dependency graph?

You're right, this is a bit of a misnomer 🫣

`renovate-graph` was chosen as a name at the project's creation, as it sounded like it worked for the project.

In retrospect, this wasn't the best name, but it's where we are right now!

It may be that at some point in the future, we'll start producing a full graph of dependency data, or maybe this project will be renamed. Let's see!

## Usage

`renovate-graph` is available as an executable package, and can be installed with:

```sh
# to invoke with `npm exec renovate-graph`
npm i @jamietanna/renovate-graph
# to invoke with `renovate-graph`
npm i -g @jamietanna/renovate-graph
# to invoke with `renovate-graph`
npx @jamietanna/renovate-graph@latest
```

### A single repository

To run `renovate-graph` against a single repository, and then output a JSON blob, you can run:

```sh
renovate-graph --token $GITHUB_COM_TOKEN jamietanna/jamietanna
```

This will create the file `out/github-jamietanna-jamietanna.json`.

### Autodiscovery (with a filter)

You can also leverage Renovate's [`autodiscover` functionality](https://docs.renovatebot.com/self-hosted-configuration/#autodiscover) to automagically discover repositories.

**NOTE**: using `autodiscover` requires **Write-access** to the underlying repository. This due to how `renovate-graph` leverages Renovate's underlying autodiscovery mechanism, which requires write access to create branches.

You can run the following to find any repositories under `github.com/jamietanna/` that the specified `GITHUB_COM_TOKEN` can access:

```sh
renovate-graph --token $GITHUB_COM_TOKEN --autodiscover --autodiscover-filter 'jamietanna/*'
```

This will create the files `out/github-jamietanna-jamietanna`, `out/github-jamietanna-disARM`, etc.

### Using the `local` platform

Since Renovate 35.76.0, it's been possible to use Renovate's [local platform](https://docs.renovatebot.com/modules/platform/local/) to run Renovate against a local directory, without requiring a full Git checkout.

From a directory - possibly without a `.git` folder - you can then run:

```sh
env RG_LOCAL_PLATFORM=gitlab RG_LOCAL_ORGANISATION=jamietanna RG_LOCAL_REPO=jamietanna renovate-graph --platform local
```

This requires the following environment variables to ensure the metadata is correct:

- `RG_LOCAL_PLATFORM`: The platform that this local directory is for
- `RG_LOCAL_ORGANISATION`: The organisation that this local directory is for. Can include a `/` if a nested organisation
- `RG_LOCAL_REPO`: The repository name that this local directory is for

#### Resolving `local>` presets

If you point Renovate to a repository with the [`local>` preset](https://docs.renovatebot.com/config-presets/#local-presets), such as the below, then Renovate will end up erroring:

```json
{
  "$schema": "https://docs.renovatebot.com/renovate-schema.json",
  "extends": [
    "local>oapi-codegen/renovate-config"
  ]
}
```

This is because Renovate doesn't understand how to resolve this.

However, `renovate-graph` approaches this differently.

Instead, because `renovate-graph` knows that i.e. `RG_LOCAL_PLATFORM=github`, it will then resolve that configuration from:

```diff
 {
   "$schema": "https://docs.renovatebot.com/renovate-schema.json",
   "extends": [
-    "local>oapi-codegen/renovate-config"
+    "github>oapi-codegen/renovate-config"
   ]
 }
```

This will allow you to resolve these presets, while still processing in the `local` platform.

### Specifying a directory for the data

If you wish to export your data to a separate location, you can specify the `OUT_DIR` environment variable to either a relative or absolute path, such as:

```sh
env OUT_DIR=../data renovate-graph --token $GITHUB_COM_TOKEN jamietanna/jamietanna
```

This will then create the file `../data/github-jamietanna-jamietanna.json`.

### Running as a GitHub App

To simplify the means to authenticate, as well as more easily discovering repositories that you wish to retrieve data from, this has support for running as a GitHub App.

The App needs to be created with the following permissions:

- Contents: `Read only`
- Issues: `Read only`
- Metadata: `Read only`

Then, when running this, you will need to set the following environment variables:

```sh
export RG_GITHUB_APP_ID='...'
export RG_GITHUB_APP_KEY='-----BEGIN RSA PRIVATE KEY-----\nMIIEowIBAAKCAQEA4XmSD...'
export RENOVATE_USERNAME='renovate[bot]'
export RENOVATE_GIT_AUTHOR='Renovate Bot <29139614+renovate[bot]@users.noreply.github.com>'

# may only be required if you're using this as a library, not an executable
export RG_GITHUB_APP_INSTALLATION_ID='...'
```

### Additional configuration

As `renovate-graph` is built on top of Renovate, any of the [self-hosted](https://docs.renovatebot.com/self-hosted-configuration/) or [repository](https://docs.renovatebot.com/configuration-options/) configuration options that apply to Renovate, also apply for `renovate-graph`.

As noted above, there are some settings that will be ignored by `renovate-graph`, for instance [`enabledManagers`](https://docs.renovatebot.com/configuration-options/#enabledmanagers) or [`packageRules`](https://docs.renovatebot.com/configuration-options/#packagerules) that disable a package.

There are also the following configuration options for further operability:

#### `RENOVATE_REQUIRE_CONFIG=optional` / `--require-config=optional`

The Renovate configuration, [`RENOVATE_REQUIRE_CONFIG`](https://docs.renovatebot.com/self-hosted-configuration/#requireconfig) defaults to `optional` with `renovate-graph`, but can be overridden.

If any Renovate configuration is set in a repository, `renovate-graph` will prevent the _disabling_ of managers, which could limit the amount of data that can be retrieved by `renovate-graph`, while allowing the use of custom `regexManagers` or otherwise, we explicitly ignore `enabledManagers` and `packageRules` in a given repo.

#### `RENOVATE_DRY_RUN=extract` / `--dry-run=extract`

By default, `renovate-graph` will run under [`RENOVATE_DRY_RUN=extract`](https://docs.renovatebot.com/self-hosted-configuration/#dryrun), which only determines what dependencies are currently in the repository being scanned.

This means `renovate-graph` will not detect if there are any pending package updates.

More importantly, this also can lead to not knowing the current version a dependency is resolved at. For instance, if a dependency is pinned to `^35.x`, `renovate-graph` cannot determine that this would resolve to i.e. `35.159.7` until running in `RENOVATE_DRY_RUN=lookup`, or `RG_INCLUDE_UPDATES=true`.

This can be modified by overriding this variable to `RENOVATE_DRY_RUN=lookup`, or setting `RG_INCLUDE_UPDATES=true`.

If `RENOVATE_DRY_RUN` is not set, `renovate-graph` will run under its default, `RENOVATE_DRY_RUN=extract`.

Using `RENOVATE_DRY_RUN=full` is treated as `RENOVATE_DRY_RUN=lookup`.

#### `RG_INCLUDE_UPDATES=false`

By default, `renovate-graph` will run under [`RENOVATE_DRY_RUN=extract`](https://docs.renovatebot.com/self-hosted-configuration/#dryrun), which only determines what dependencies are currently in the repository being scanned.

This means `renovate-graph` will not detect if there are any pending package updates.

More importantly, this also can lead to not knowing the current version a dependency is resolved at. For instance, if a dependency is pinned to `^35.x`, `renovate-graph` cannot determine that this would resolve to i.e. `35.159.7` until running in `RENOVATE_DRY_RUN=lookup`, or `RG_INCLUDE_UPDATES=true`.

This can be modified by overriding `RENOVATE_DRY_RUN=lookup`, or setting `RG_INCLUDE_UPDATES=true`.

As well as parsing the dependency tree, also receive the list of updates that Renovate can see across your dependencies. This is opt-in as it can lead to considerably slower executions of `renovate-graph` due to needing to hit the network much higher for dependency updates.

This is synonymous with `RENOVATE_DRY_RUN=lookup`.

#### `RG_DELETE_CLONED_REPOS=false`

After checking the dependency data for each repo, immediately delete it. This will slow execution of renovate-graph, but will allow running when using large repositories or against a large number of repositories and organisations that could lead to exhaustion of disk space.

This generally isn't necessary, unless you regularly hit disk space issues.

#### `RG_INCLUDE_LOCKFILES=false`

Parse any lockfiles that Renovate discovers for any dependencies that may be referenced.

This is off by default as these aren't _necessarily_ dependencies being used, and can significantly bloat the dependency import.

#### `RG_DO_NOT_FETCH_NON_LOCAL_PRESETS=false`

When running in `--platform local` mode, unlike `renovate`, `renovate-graph` will attempt to discover the presets that may be defined on a platform, i.e. `github>...` or using the `local>` syntax. However, this may cause issues i.e. if you require specifying your self-hosted instance's [`--endpoint`](https://docs.renovatebot.com/self-hosted-configuration/#endpoint), and so you can disable this functionality by setting `RG_DO_NOT_FETCH_NON_LOCAL_PRESETS=true`.

#### `RG_ALLOW_UNPATCHED_DEPENDENCIES=false`

`renovate-graph` patches the underlying Renovate dependency using [`patch-package`](https://www.npmjs.com/package/patch-package), applying a number of changes that tunes certain functionality better for `renovate-graph`.

If `renovate-graph` detects that the patching process hasn't successfully worked - for instance, if you're running `pnpx @jamietanna/renovate-graph` (which is not currently supported) - then `renovate-graph` will fail to continue.

This prevents the execution of `renovate-graph` without important patches, which could lead to `renovate-graph` not processing repositories correctly.

However, it's possible to override this with `RG_ALLOW_UNPATCHED_DEPENDENCIES=true` to allow this error to be ignored. This **isn't recommended**, and should only be considered if you're aware of the functionality that is missing if you were to not have these patches applied.

### Using `renovate-graph` as a library

👷🏼 👷🏼 👷🏼 👷🏼

This section is under construction!

👷🏼 👷🏼 👷🏼 👷🏼

## License

As this is heavily modified code from Renovate itself, this project is licensed in the same was as Renovate - AGPL-3.0.