feat(tokens): add secure token generator/validator with entropy + length caps by hyp3rd · Pull Request #40 · hyp3rd/sectools

hyp3rd · 2026-01-11T20:00:03Z

Introduce pkg/tokens with a configurable Generator and Validator:

Encodings: base64url (default) and hex.
Defaults: min entropy 128 bits; max token length 4096.
Options: min bytes, max length, encoding, min entropy.
Validation checks: empty, too short/long, invalid format, insufficient entropy.
Errors: ErrInvalidTokenConfig, ErrTokenEmpty, ErrTokenTooLong, ErrTokenTooShort, ErrTokenInvalid, ErrTokenInsufficientEntropy.

Tests:

Cover base64url/hex generation & validation, length bounds, min bytes, entropy checks, and whitespace handling.

Docs:

README and docs/usage.md reference pkg/tokens and usage examples.
docs/security-checklist.md adds “Random Tokens” guidance.

Chore:

Add cspell terms (“base64url”, “entropy”).
Add .github/FUNDING.yml.

…gth caps Introduce pkg/tokens with a configurable Generator and Validator: - Encodings: base64url (default) and hex. - Defaults: min entropy 128 bits; max token length 4096. - Options: min bytes, max length, encoding, min entropy. - Validation checks: empty, too short/long, invalid format, insufficient entropy. - Errors: ErrInvalidTokenConfig, ErrTokenEmpty, ErrTokenTooLong, ErrTokenTooShort, ErrTokenInvalid, ErrTokenInsufficientEntropy. Tests: - Cover base64url/hex generation & validation, length bounds, min bytes, entropy checks, and whitespace handling. Docs: - README and docs/usage.md reference pkg/tokens and usage examples. - docs/security-checklist.md adds “Random Tokens” guidance. Chore: - Add cspell terms (“base64url”, “entropy”). - Add .github/FUNDING.yml.

Copilot

Pull request overview

This PR introduces a new pkg/tokens package that provides cryptographically secure random token generation and validation with configurable entropy requirements and length limits. The implementation supports both base64url (default) and hexadecimal encoding, with comprehensive validation including entropy checks, length bounds, and whitespace rejection.

Changes:

Added pkg/tokens package with Generator and Validator types supporting configurable encodings (base64url/hex), entropy requirements, and length constraints
Added comprehensive error types for token validation failures
Updated documentation in README.md, docs/usage.md, and docs/security-checklist.md to reference the new token functionality
Added cspell terms and FUNDING.yml configuration

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 7 comments.

Show a summary per file

File	Description
pkg/tokens/tokens.go	Core implementation of token generation and validation with entropy/length controls
pkg/tokens/errors.go	Error definitions for token operations
pkg/tokens/doc.go	Package documentation
pkg/tokens/tokens_test.go	Test coverage for token generation and validation
docs/usage.md	Added token package documentation and usage examples
docs/security-checklist.md	Added security guidance for random tokens
README.md	Added token generation example to main documentation
cspell.json	Added "base64url" and "entropy" to spell check dictionary
.yamllint	Added FUNDING.yml to ignore list
.github/FUNDING.yml	Added GitHub Sponsors funding configuration

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-01-11T20:05:36Z

pkg/tokens/tokens.go

+	if len(decoded)*bitsPerByte < v.opts.minEntropyBits {
+		return nil, ErrTokenInsufficientEntropy
+	}


The entropy validation logic assumes that all bytes in the decoded token contribute to entropy. However, this doesn't account for the actual randomness of the bytes. A token filled with zero bytes (make([]byte, 16)) would pass the entropy check even though it has no actual entropy. While this might be acceptable for validation of pre-generated tokens, the validation doesn't truly measure entropy - it only measures length. Consider renaming the validation to reflect this (e.g., minDecodedBytes instead of minEntropyBits) or add documentation clarifying that this checks theoretical maximum entropy, not actual entropy.

Copilot · 2026-01-11T20:05:36Z

README.md

+ token, _ := generator.Generate()
+ _, _ = validator.Validate(token)


The example code silently ignores errors from Generate() and Validate() using blank identifier (_). This sets a poor example for users, especially in a security-focused library. Even in example code, errors from token generation and validation should be properly handled or at least demonstrated with proper error handling patterns.

Suggested change

token, _ := generator.Generate()

_, _ = validator.Validate(token)

token, err := generator.Generate()

if err != nil {

panic(err)

}

if _, err := validator.Validate(token); err != nil {

panic(err)

}

pkg/tokens/tokens_test.go

pkg/tokens/tokens.go

Copilot · 2026-01-11T20:05:38Z

pkg/tokens/doc.go

@@ -0,0 +1,2 @@
+// Package tokens provides safe random token generation and validation.


The package documentation is minimal. Consider expanding it to include key features and a basic usage example, such as:

Supported encodings (base64url, hex)

Default security parameters (128-bit entropy, 4096 max length)

Basic usage pattern showing NewGenerator/Generate and NewValidator/Validate

Thread-safety guarantees

This would help users quickly understand the package capabilities without navigating to separate documentation files.

Suggested change

// Package tokens provides safe random token generation and validation.

// Package tokens provides safe random token generation and validation.

//

// Features

//

// - Cryptographically secure random token generation suitable for use in

// session identifiers, CSRF tokens, API keys, and similar use cases.

// - Pluggable encodings, including base64url and hexadecimal (hex).

// - Built-in validation helpers to parse and verify previously generated

// tokens.

//

// Defaults

//

// Unless otherwise configured, generators created by this package:

//

// - Use 128 bits of entropy for each generated token.

// - Enforce a maximum token length of 4096 bytes to guard against

// resource-exhaustion attacks.

//

// Basic usage

//

// The typical workflow is:

//

// 1. Construct a Generator with NewGenerator.

// 2. Call Generate to create new random tokens.

// 3. Construct a Validator with NewValidator.

// 4. Call Validate to check incoming tokens.

//

// Example:

//

// package main

//

// import (

// "fmt"

//

// "example.com/project/pkg/tokens"

// )

//

// func main() {

// // Create a generator using default settings (128-bit entropy,

// // base64url encoding, 4096-byte maximum length).

// gen := tokens.NewGenerator()

//

// // Generate a new token.

// token, err := gen.Generate()

// if err != nil {

// panic(err)

// }

// fmt.Println("generated token:", token)

//

// // Create a validator using compatible settings.

// v := tokens.NewValidator()

//

// // Validate a token value received from a client.

// if err := v.Validate(token); err != nil {

// fmt.Println("invalid token:", err)

// return

// }

// fmt.Println("token is valid")

// }

//

// Concurrency

//

// Unless documented otherwise on specific types, values returned by

// NewGenerator and NewValidator are safe for concurrent use by multiple

// goroutines. This allows a single generator or validator instance to be

// shared across handlers in high-concurrency servers.

Copilot · 2026-01-11T20:05:38Z

pkg/tokens/tokens_test.go

+package tokens
+
+import (
+	"encoding/base64"
+	"testing"
+)
+
+func TestTokenGenerateAndValidateBase64(t *testing.T) {
+	generator, err := NewGenerator()
+	if err != nil {
+		t.Fatalf("expected generator, got %v", err)
+	}
+
+	validator, err := NewValidator()
+	if err != nil {
+		t.Fatalf("expected validator, got %v", err)
+	}
+
+	token, err := generator.Generate()
+	if err != nil {
+		t.Fatalf("expected token, got %v", err)
+	}
+
+	decoded, err := validator.Validate(token)
+	if err != nil {
+		t.Fatalf("expected token valid, got %v", err)
+	}
+
+	if len(decoded) < 16 {
+		t.Fatalf("expected at least 16 bytes, got %d", len(decoded))
+	}
+}
+
+func TestTokenGenerateHex(t *testing.T) {
+	generator, err := NewGenerator(WithTokenEncoding(TokenEncodingHex))
+	if err != nil {
+		t.Fatalf("expected generator, got %v", err)
+	}
+
+	validator, err := NewValidator(WithTokenEncoding(TokenEncodingHex))
+	if err != nil {
+		t.Fatalf("expected validator, got %v", err)
+	}
+
+	token, err := generator.Generate()
+	if err != nil {
+		t.Fatalf("expected token, got %v", err)
+	}
+
+	decoded, err := validator.Validate(token)
+	if err != nil {
+		t.Fatalf("expected token valid, got %v", err)
+	}
+
+	if len(decoded) < 16 {
+		t.Fatalf("expected at least 16 bytes, got %d", len(decoded))
+	}
+}
+
+func TestTokenValidateMaxLength(t *testing.T) {
+	validator, err := NewValidator(
+		WithTokenMaxLength(4),
+		WithTokenMinEntropyBits(8),
+	)
+	if err != nil {
+		t.Fatalf("expected validator, got %v", err)
+	}
+
+	_, err = validator.Validate("aaaaa")
+	if err != ErrTokenTooLong {
+		t.Fatalf("expected ErrTokenTooLong, got %v", err)
+	}
+}
+
+func TestTokenValidateInsufficientEntropy(t *testing.T) {
+	validator, err := NewValidator(WithTokenMinEntropyBits(128))
+	if err != nil {
+		t.Fatalf("expected validator, got %v", err)
+	}
+
+	short := base64.RawURLEncoding.EncodeToString(make([]byte, 8))
+	_, err = validator.Validate(short)
+	if err != ErrTokenInsufficientEntropy {
+		t.Fatalf("expected ErrTokenInsufficientEntropy, got %v", err)
+	}
+}
+
+func TestTokenValidateMinBytes(t *testing.T) {
+	validator, err := NewValidator(WithTokenMinBytes(32))
+	if err != nil {
+		t.Fatalf("expected validator, got %v", err)
+	}
+
+	short := base64.RawURLEncoding.EncodeToString(make([]byte, 16))
+	_, err = validator.Validate(short)
+	if err != ErrTokenTooShort {
+		t.Fatalf("expected ErrTokenTooShort, got %v", err)
+	}
+}
+
+func TestTokenValidateWhitespace(t *testing.T) {
+	validator, err := NewValidator()
+	if err != nil {
+		t.Fatalf("expected validator, got %v", err)
+	}
+
+	_, err = validator.Validate("token with space")
+	if err != ErrTokenInvalid {
+		t.Fatalf("expected ErrTokenInvalid, got %v", err)
+	}
+}


The test suite is missing coverage for several important scenarios:

Test for empty token validation (ErrTokenEmpty)

Test for invalid encoding formats (invalid base64/hex strings)

Test for GenerateBytes() method

Test for invalid token configuration scenarios (negative values, conflicting options)

Test to ensure generated tokens are actually random (multiple generations should produce different tokens)

Test for edge cases like WithTokenEncoding with invalid encoding value

Given that other packages in this repository have comprehensive test coverage, these scenarios should be tested to maintain consistency.

@copilot open a new pull request to apply changes based on this feedback

- Introduce pkg/encoding with helpers and typed errors - Base64/Hex encode/decode with length checks (ErrBase64TooLong/Invalid/Empty, ErrHexTooLong/Invalid) - JSON encode/decode + reader with trailing-data detection and max-bytes/unknown-fields options (ErrJSONTooLarge/Invalid/TrailingData) - Add unit tests covering success and failure paths - Update docs (usage, security checklist) to document encoding guidance and limits - Add dependency: github.com/goccy/go-json v0.10.5 - Adjust token tests to use errors.Is and updated error handling Why: establish safe, bounded primitives for encoding/decoding, reduce input-parsing risk, and standardize error semantics across packages.

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Copilot · 2026-01-11T23:11:50Z

@hyp3rd I've opened a new pull request, #41, to work on those changes. Once the pull request is ready, I'll request review from you.

Copilot AI review requested due to automatic review settings January 11, 2026 20:00

Copilot started reviewing on behalf of hyp3rd January 11, 2026 20:00 View session

Copilot AI reviewed Jan 11, 2026

View reviewed changes

hyp3rd and others added 4 commits January 12, 2026 00:08

Update pkg/tokens/tokens.go

b918b24

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Update pkg/tokens/tokens.go

0cd9f25

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Update pkg/tokens/tokens_test.go

c4c8aeb

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Copilot AI mentioned this pull request Jan 11, 2026

Add comprehensive test coverage for token generator/validator #41

Merged

hyp3rd merged commit dce98b2 into main Jan 11, 2026
13 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(tokens): add secure token generator/validator with entropy + length caps#40

feat(tokens): add secure token generator/validator with entropy + length caps#40
hyp3rd merged 5 commits intomainfrom
feat/tokens

hyp3rd commented Jan 11, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Jan 11, 2026

Uh oh!

Copilot AI Jan 11, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI Jan 11, 2026

Uh oh!

Copilot AI Jan 11, 2026

Uh oh!

hyp3rd Jan 11, 2026

Uh oh!

Copilot AI commented Jan 11, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		token, _ := generator.Generate()
		_, _ = validator.Validate(token)

		@@ -0,0 +1,2 @@
		// Package tokens provides safe random token generation and validation.

-// Package tokens provides safe random token generation and validation.
+// Package tokens provides safe random token generation and validation.
+//
+// Features
+//
+//   - Cryptographically secure random token generation suitable for use in
+//     session identifiers, CSRF tokens, API keys, and similar use cases.
+//   - Pluggable encodings, including base64url and hexadecimal (hex).
+//   - Built-in validation helpers to parse and verify previously generated
+//     tokens.
+//
+// Defaults
+//
+// Unless otherwise configured, generators created by this package:
+//
+//   - Use 128 bits of entropy for each generated token.
+//   - Enforce a maximum token length of 4096 bytes to guard against
+//     resource-exhaustion attacks.
+//
+// Basic usage
+//
+// The typical workflow is:
+//
+//   1. Construct a Generator with NewGenerator.
+//   2. Call Generate to create new random tokens.
+//   3. Construct a Validator with NewValidator.
+//   4. Call Validate to check incoming tokens.
+//
+// Example:
+//
+//   package main
+//
+//   import (
+//       "fmt"
+//
+//       "example.com/project/pkg/tokens"
+//   )
+//
+//   func main() {
+//       // Create a generator using default settings (128-bit entropy,
+//       // base64url encoding, 4096-byte maximum length).
+//       gen := tokens.NewGenerator()
+//
+//       // Generate a new token.
+//       token, err := gen.Generate()
+//       if err != nil {
+//           panic(err)
+//       }
+//       fmt.Println("generated token:", token)
+//
+//       // Create a validator using compatible settings.
+//       v := tokens.NewValidator()
+//
+//       // Validate a token value received from a client.
+//       if err := v.Validate(token); err != nil {
+//           fmt.Println("invalid token:", err)
+//           return
+//       }
+//       fmt.Println("token is valid")
+//   }
+//
+// Concurrency
+//
+// Unless documented otherwise on specific types, values returned by
+// NewGenerator and NewValidator are safe for concurrent use by multiple
+// goroutines. This allows a single generator or validator instance to be
+// shared across handlers in high-concurrency servers.

Uh oh!

Conversation

hyp3rd commented Jan 11, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Jan 11, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 11, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI Jan 11, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 11, 2026

Choose a reason for hiding this comment

Uh oh!

hyp3rd Jan 11, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI commented Jan 11, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants