Secret Detection Service

Secret Detection service is primarily responsible for detecting secrets in the given input payloads with RPC methods as the communication interface served via gRPC. This service will initially be invoked by Rails monolith when performing access checks for Git Push event, and eventually extended for the other usecases too.

Reference Issue: https://gitlab.com/groups/gitlab-org/-/epics/13792

Tools and Framework

  • Ruby 3.2.5
  • gRPC framework for serving RPC requests

Feature Distribution

In addition to offering the feature as an gRPC service, this project also includes the provision for distributing the same feature into a Ruby Gem. This provision was added to fulfil certain limitations. Here's the illustration representing the approach:

Feature Distribution

Project Layout

├── .runway
│   ├── runway.yml                       # Runway configuration file for Production environment
│   ├── runway-staging.yml               # Runway configuration file for Staging environment
│   └── env-*.yml                        # Environment vars for the respective environments. Uses vault for secrets
├── bin
│   └── start_server                     # gRPC server initiator that loads the server configuration file
├── config
│   └── log.rb                           # Logger configuration
├── ci-templates
│   ├── build.yml                        # CI jobs for building container image and ruby gems
│   ├── test.yml                         # CI jobs for running tests
│   └── runway.yml                       # Runway-downstream trigger and overriding some of runway's downstream CI jobs
├── lib
│   └── gitlab
│       └── secret_detection  
│           ├── version.rb               # Secret Detection Gem release version
│           ├── core/..                  # Secret detection logic (most of it pulled from existing gem)
│           └── grpc 
│               ├── generated/..         # gRPC generated files and secret detection gRPC service
│               └── scanner_service.rb   # Secret detection gRPC service implementation
├── examples
│   └── sample-client/..                 # Sample Ruby RPC client that connects with gRPC server and calls RPC scan
├── proto
│   └── secret_detection.proto           # Service Definition file containing gRPC request and response interface
├── server
│   ├── interceptors/..                  # gRPC server-side interceptors like Auth, Log etc.
│   └── server.rb                        # gRPC server file with configuration
├── spec/..                              # Rspec tests and related test helpers
├── gitlab-secret_detection.gemspec      # Gemspec file for Ruby Gem
├── Dockerfile                           # Dockerfile for running gRPC server
└── Makefile                             # All the CLI commands placed here

Makefile commands

Usage make <command>

Command Description
install Installs ruby gems in the project using Ruby bundler
lint_fix Fixes all the fixable Rubocop lint offenses
gem_clean Cleans existing gem file(if any) generated through gem build process
gem_build Builds Ruby gem file wrapping secret detection logic (lib directory)
generate_proto Generates ruby(.rb) files for the Protobud Service Definition files(.proto)
grpc_docker_build Builds a docker container image for gRPC server
grpc_docker_serve Runs gRPC server via docker container listening on port 8080. Run grpc_docker_build make command before running this command.
grpc_serve Runs gRPC server on the CLI listening on port 50001. Run install make command before running this command.
run_core_tests Runs RSpec tests for Secret Detection core logic
run_grpc_tests Runs RSpec tests for Secret Detection gRPC endpoints
run_all_tests Runs all the RSpec tests in the project

Generating a ruby gem

In the project directory, run make gem command in the terminal that builds a ruby gem(ex: secret_detection-0.1.0.gem) in the root of the project directory.

Server

This project currently runs on gRPC server.

Service Definitions

Running the server locally

Pre-requisite: gRPC installed on your system (brew install grpc)

server mode command Listening port
gRPC CLI make grpc_serve 50001
gRPC Docker make grpc_docker_build && make grpc_docker_serve 8080

gRPC server port can be configured via RPC_SERVER_PORT environment variable

Calling gRPC endpoints from terminal

Pre-requisite:

  • gRPC installed on your system (via brew install grpc)
  • gRPC client like Postman or you can install grpcurl (via brew install grpcurl)

Authentication:

The RPC service uses a basic form of token-based authentication. When invoking an RPC request on the client side, we need to append an x-sd-auth RPC Header whose value is dependent on the environment where the server is running.

The auth token value when the server is running in:

  • Localhost: 12345
  • Staging: env/staging/service/secret-detection/API_AUTH_TOKEN path in Vault
  • Production: env/production/service/secret-detection/API_AUTH_TOKEN path in Vault

Note that only Secret Detection RPC requests are guarded with authentication.

More details can be found in this issue.

Health Check

RPC Method: grpc.health.v1.Health/Check

Example ```shell $ grpcurl -plaintext -d '"service":"gitlab"service":"gitlab.secret_detection"service":"gitlab.secret_detection.Scanner"' localhost:50001 grpc.health.v1.Health/Check ``` You should see the following response as a result: ```shell { "status": "SERVING" } ```

Secret Detection Scan (Unary RPC Call)

RPC Method: gitlab.secret_detection.Scanner/Scan Default Timeout(configurable): per-request: 180 seconds, per-payload: 30 seconds

Example: Basic ```shell $ grpcurl -d @ \ localhost:50001 \ -rpc-header 'x-sd-auth:12345' \ gitlab.secret_detection.Scanner/Scan <
Example: Using Allowlist and Tags ```shell $ grpcurl -d @ \ localhost:50001 \ -rpc-header 'x-sd-auth:12345' \ gitlab.secret_detection.Scanner/Scan <

Secret Detection Scan (Bi-directional RPC Streaming)

RPC Method: gitlab.secret_detection.Scanner/ScanStream Default Timeout(configurable): per-request: 180 seconds, per-payload: 30 seconds

Bi-directional RPC streaming allows the server to receive a continuous stream of requests and respond with a corresponding stream of responses as each request is processed. Unlike unary RPC calls, where requests are sent once and the channel closes after a single response, bi-directional streaming keeps the RPC channel open to handle ongoing requests and responses, enabling continuous communication between the client and server.

Example To try this out, we will accept the request input from STDIN. As on when you provide request input json in the terminal, you should see a corresponding server response. The connection will continue to remain open for accepting requests unless you explicitly close the connection with `Ctrl+C`. Here's the command to start an RPC streaming channel on the terminal: ```shell $ grpcurl -d @ localhost:50001 -rpc-header 'x-sd-auth:12345' gitlab.secret_detection.Scanner/ScanStream ``` Once the channel is open, enter the following sample request input in the terminal: ```json { "payloads": [ { "id": "94283", "data": "glpat-12345123451234512345" } ] } ``` you should immediately see the Server's response like below: ```json { "results": [ { "payload_id": "94283", "status": "STATUS_FOUND", "type": "gitlab_personal_access_token", "description": "GitLab Personal Access Token", "line_number": 1 } ], "status": "STATUS_FOUND" } ``` You may continue to provide more request inputs to the opened RPC channel to receive corresponding server responses. Features like Allowlisting, Tag filtering works same as outlined under Unary call section.

NOTE:

  • In case you're running the local server via Docker, replace localhost:50001 with localhost:8080 in the example grpcurl commands.
  • In case you want to access staging server, replace localhost:50001 with secret-detection.staging.runway.gitlab.net:443

How to invoke the server from the code (as an RPC Client)

There is a sample Ruby RPC Client in the project that contains following code for reference:

  • Setting up the connection with the server (both local and remote server with SSL)
  • Invoke Secret Detection Unary call
  • Invoke Secret Detection bi-directional stream call

Run ruby examples/sample-client/sample_client.rb on your terminal to run the sample RPC client calling both unary and streaming RPC Scan methods.

Benchmark

RPC service is benchmarked using ghz, a powerful CLI-based tool for load testing and benchmarking gRPC services. More details added here.

Project Status

Secret Detection service's status can be tracked here: https://gitlab.com/gitlab-org/gitlab/-/issues/467531

Changes made in the secret detection logic that were previously not present in the Gem