Account realloc issue

[lgalabru]

In the process of stressing Surfpool with token-2022 - It's unclear if I'm doing something wrong. I'm trying to initialize a token with the following script:

#!/bin/bash

# =============================================================================
# TOK Token Creation Script
# =============================================================================
# This script creates a Token-2022 token with:
#   - Transfer Hook extension (custom logic on every transfer)
#   - Metadata extension (on-chain token name, symbol, URI)
# =============================================================================

set -e  # Exit on any error

# -----------------------------------------------------------------------------
# Configuration
# -----------------------------------------------------------------------------
TRANSFER_HOOK_PROGRAM="BpZKE38oJmRtVLhMD6zdL9YNwH5HP63gcYac9djsVBpr"
TOKEN_NAME="TOK"
TOKEN_SYMBOL="TOK"
TOKEN_URI=""  # Optional: URL to off-chain metadata JSON
MINT_AMOUNT="100"
ENABLE_METADATA=false  # Set to true for devnet/mainnet, false for local simnet

# -----------------------------------------------------------------------------
# Step 1: Create the Token Mint with Metadata
# -----------------------------------------------------------------------------
# This creates the MINT ACCOUNT - the central registry for the token.
#
# Account: MINT (e.g., 946jfDA2UEeghgCYPomwiZjmVXCRjiiMfrwd6U87LD53)
#   - Owner: Token-2022 Program (TokenzQdBNbLqP5VEhdkAS6EPFLC1PHnBqCXEpPxuEb)
#   - Purpose: Stores token configuration (decimals, supply, authorities)
#   - Extensions enabled:
#     1. Transfer Hook: Points to our program that runs on every transfer
#     2. Metadata: Allows storing name/symbol/uri directly in the mint
#   - Authorities set to your wallet:
#     - Mint Authority: Can mint new tokens
#     - Transfer Hook Authority: Can update the hook program
#     - Metadata Update Authority: Can update token metadata
#
# The mint account is a PDA-like structure that:
#   - Tracks total supply of tokens in circulation
#   - Defines decimal precision (9 decimals = 1 token = 1,000,000,000 base units)
#   - References the transfer hook program to invoke on transfers
#
# IMPORTANT: We initialize metadata in the same command to avoid reallocation
# issues. The --initialize-metadata flag creates the mint AND initializes
# metadata atomically, ensuring proper space allocation.
# -----------------------------------------------------------------------------
echo "Creating token mint with transfer hook..."

# Build create-token command
CREATE_CMD="spl-token create-token --program-2022 --transfer-hook $TRANSFER_HOOK_PROGRAM"
if [ "$ENABLE_METADATA" = true ]; then
    CREATE_CMD="$CREATE_CMD --enable-metadata"
fi

MINT_OUTPUT=$(eval "$CREATE_CMD" 2>&1)
echo "$MINT_OUTPUT"

# Extract mint address from output
MINT_ADDRESS=$(echo "$MINT_OUTPUT" | grep "Address:" | awk '{print $2}')

if [ -z "$MINT_ADDRESS" ]; then
    echo "Error: Failed to extract mint address"
    exit 1
fi

echo "Mint address: $MINT_ADDRESS"

# -----------------------------------------------------------------------------
# Step 2: Initialize Token Metadata (optional)
# -----------------------------------------------------------------------------
# This writes metadata directly into the mint account's extension data.
#
# No new account is created - this modifies the MINT ACCOUNT to include:
#   - Name: Human-readable token name (e.g., "TOK")
#   - Symbol: Trading symbol (e.g., "TOK")
#   - URI: Link to off-chain JSON with additional metadata (icon, description)
#
# This is different from Metaplex metadata which creates a separate PDA.
# Token-2022 metadata lives directly in the mint account, reducing complexity.
#
# NOTE: This requires account reallocation which may fail on local simnet.
# Set ENABLE_METADATA=true for devnet/mainnet.
# -----------------------------------------------------------------------------
if [ "$ENABLE_METADATA" = true ]; then
    echo "Initializing token metadata..."
    spl-token initialize-metadata "$MINT_ADDRESS" "$TOKEN_NAME" "$TOKEN_SYMBOL" "$TOKEN_URI" --program-2022
else
    echo "Skipping metadata initialization (ENABLE_METADATA=false)"
fi

# -----------------------------------------------------------------------------
# Step 3: Create Associated Token Account (ATA)
# -----------------------------------------------------------------------------
# This creates a TOKEN ACCOUNT to hold the tokens for your wallet.
#
# Account: ASSOCIATED TOKEN ACCOUNT (e.g., 46fTK3QWahnGrTYsZcSQR7N55QQiFuyH9kC1au5TBUvT)
#   - Owner: Token-2022 Program
#   - Purpose: Holds token balance for a specific wallet + mint combination
#   - Authority: Your wallet (can transfer tokens out)
#
# The ATA is a Program Derived Address (PDA) deterministically generated from:
#   - Your wallet address
#   - The token mint address
#   - The Token-2022 program ID
#   - The Associated Token Account program ID
#
# Formula: PDA = findProgramAddress([wallet, token_program, mint], ATA_program)
#
# This ensures:
#   - Each wallet has exactly ONE token account per mint
#   - Anyone can derive the address without on-chain lookup
#   - Tokens can be sent to a wallet even if the ATA doesn't exist yet
# -----------------------------------------------------------------------------
echo "Creating associated token account..."
spl-token create-account "$MINT_ADDRESS" --program-2022

# -----------------------------------------------------------------------------
# Step 4: Mint Initial Supply
# -----------------------------------------------------------------------------
# This increases the token supply and credits tokens to your ATA.
#
# No new account is created. This transaction:
#   1. Updates MINT ACCOUNT: Increases total supply counter
#   2. Updates your ATA: Increases token balance
#
# Requires: Mint Authority signature (your wallet, set during mint creation)
#
# Amount: 100 tokens = 100,000,000,000 base units (with 9 decimals)
# -----------------------------------------------------------------------------
echo "Minting initial supply..."
spl-token mint "$MINT_ADDRESS" "$MINT_AMOUNT" --program-2022

# -----------------------------------------------------------------------------
# Step 5: Initialize Extra Account Metas (for Transfer Hook)
# -----------------------------------------------------------------------------
# This creates the EXTRA ACCOUNT METAS account - required for transfer hooks.
#
# Account: EXTRA ACCOUNT METAS PDA
#   - Address: PDA derived from seeds ["extra-account-metas", mint_address]
#   - Owner: Your Transfer Hook Program
#   - Purpose: Tells the Token-2022 program which additional accounts
#              to include when invoking your transfer hook
#
# When a transfer happens:
#   1. Token-2022 reads this account to discover required extra accounts
#   2. Token-2022 includes those accounts in the CPI to your hook program
#   3. Your hook program can then access those accounts during transfer
#
# Without this account, transfers will fail because Token-2022 won't know
# how to call your transfer hook program correctly.
#
# NOTE: This step requires calling your Anchor program's initialize instruction.
# Uncomment and modify the following when your program has an init function:
# -----------------------------------------------------------------------------
# echo "Initializing extra account metas for transfer hook..."
# anchor run initialize-extra-account-metas -- --mint "$MINT_ADDRESS"

# -----------------------------------------------------------------------------
# Summary
# -----------------------------------------------------------------------------
echo ""
echo "============================================="
echo "Token creation complete!"
echo "============================================="
echo "Mint:     $MINT_ADDRESS"
echo "Name:     $TOKEN_NAME"
echo "Symbol:   $TOKEN_SYMBOL"
echo "Supply:   $MINT_AMOUNT tokens"
echo "Hook:     $TRANSFER_HOOK_PROGRAM"
echo "============================================="
echo ""
echo "Accounts created:"
echo "  1. Mint Account        - Central token registry with extensions"
echo "  2. Your Token Account  - ATA holding your token balance"
echo ""
echo "Next steps:"
echo "  - Initialize extra account metas PDA for transfer hook"
echo "  - Deploy and test the transfer hook program"

and seeing the following error:

Error: Client(Error { request: Some(SendTransaction), kind: RpcError(RpcResponseError { code: -32002, message: "Transaction simulation failed: Error processing Instruction 1: Failed to reallocate account data: 6 log messages:\nProgram 11111111111111111111111111111111 invoke [1]\nProgram 11111111111111111111111111111111 success\nProgram TokenzQdBNbLqP5VEhdkAS6EPFLC1PHnBqCXEpPxuEb invoke [1]\nProgram log: TokenMetadataInstruction: Initialize\nProgram TokenzQdBNbLqP5VEhdkAS6EPFLC1PHnBqCXEpPxuEb consumed 3019 of 1399850 compute units\nProgram TokenzQdBNbLqP5VEhdkAS6EPFLC1PHnBqCXEpPxuEb failed: Failed to reallocate account data", data: SendTransactionPreflightFailure(RpcSimulateTransactionResult { err: Some(InstructionError(1, InvalidRealloc)), logs: Some(["Program 11111111111111111111111111111111 invoke [1]", "Program 11111111111111111111111111111111 success", "Program TokenzQdBNbLqP5VEhdkAS6EPFLC1PHnBqCXEpPxuEb invoke [1]", "Program log: TokenMetadataInstruction: Initialize", "Program TokenzQdBNbLqP5VEhdkAS6EPFLC1PHnBqCXEpPxuEb consumed 3019 of 1399850 compute units", "Program TokenzQdBNbLqP5VEhdkAS6EPFLC1PHnBqCXEpPxuEb failed: Failed to reallocate account data"]), accounts: None, units_consumed: Some(1400000), return_data: Some(UiTransactionReturnData { program_id: "TokenzQdBNbLqP5VEhdkAS6EPFLC1PHnBqCXEpPxuEb", data: ("", Base64) }), inner_instructions: None, replacement_blockhash: None }) }) })

[Shubbu03]

from what i understand -

in simnet, spl-token initialize-metadata fails on Token-2022 mints with InvalidRealloc. This seems to be be because the litesvm-based runtime doesn’t currently support (or is misconfigured for) reallocating the mint account to grow for the metadata extension. Token-2022 works in general, but this specific realloc path isn’t handled, and there’s no test coverage to catch it.

if so, then what do you suggest we should proceed?

1. fixing it properly by enabling/fixing realloc support in litesvm
2. Add a workaround (e.g., require correct mint sizing upfront or block this flow)?

also we can add tests for tkn22 metadata init

[lgalabru]

Interesting, thanks for looking into this @Shubbu03 !
What makes you think the issue is in litesvm?

[lgalabru]

We did rush partial support for token2022, any help on battle testing the implementation with unit testing would be greatly appreciated!!

[Shubbu03]

actually, i dont have concrete proof that its due to litesvm, The InvalidRealloc error occurs when Token-2022 tries to grow the mint account during initialize_metadata, but I haven't traced exactly where the failure happens.

so i'll write a test which will reproduce this flow and then test it for further confirmation.

[Shubbu03]

We did rush partial support for token2022, any help on battle testing the implementation with unit testing would be greatly appreciated!!

sure, for this first i'll write test for metadata ext init and then debug it for InvalidRealloc issue. after that i'll also add tests for transfer hooks, transfer fees, interest bearing tkns, close operations, delegate approve/revoke

[lgalabru]

@Shubbu03 do you want to take a stab at this realloc error? Looks like some other users ran into it, we'll need to send a fix soon!

[Shubbu03]

@Shubbu03 do you want to take a stab at this realloc error? Looks like some other users ran into it, we'll need to send a fix soon!

sure, i'll look into this

[Shubbu03]

i looked again and found that basic tkn 22 operations work because they dont require acc reallocation, but to store name, uri, symbol, initilaise-metadata requires reallocation, so ig this fails in litesvm. so we can do one of these -

• pre-allocate the mint acc (padding) like:
  let metadata_len = name.len() + symbol.len() + uri.len() + padding;
let mint_size = Mint::LEN + MetadataPointer::SIZE + TokenMetadata::BASE_SIZE + metadata_len;
• load the tkn 22 program as BPF program

i can impl one of these to see which one fixes the issue, wdyt?

[MicaiahReid]

I think I have gotten to the bottom of this issue.

Recently SIMD-0129 was merged and implemented in downstream libs.

Additionally, recently Surfpool updated deps to update default SVM features, and this change included the implementation of this SIMD.

The Impact section of the SIMD outlines some changes that would need to happen in some libs to avoid this error. It seems as though others have run into this issue.

I'm not sure if the token metadata dependencies being used by the spl-token initialize-metadata command still need to be updated, or if just the spl-token binary needs to be updated, but disabling the feature in surfpool seems to unblock this issue.

@Shubbu03 Can you confirm if starting surfpool with the following flag fixes things? surfpool start --disable-feature stricter-abi-and-runtime-constraints

[Shubbu03]

I think I have gotten to the bottom of this issue.

Recently SIMD-0129 was merged and implemented in downstream libs.

Additionally, recently Surfpool updated deps to update default SVM features, and this change included the implementation of this SIMD.

The Impact section of the SIMD outlines some changes that would need to happen in some libs to avoid this error. It seems as though others have run into this issue.

I'm not sure if the token metadata dependencies being used by the spl-token initialize-metadata command still need to be updated, or if just the spl-token binary needs to be updated, but disabling the feature in surfpool seems to unblock this issue.

@Shubbu03 Can you confirm if starting surfpool with the following flag fixes things? surfpool start --disable-feature stricter-abi-and-runtime-constraints

yeah, this works. I disabled this and it working fine for me now.