Differentiable KS Guitar Synthesizer¶

Karplus-Strong DDSP acoustic guitar synthesizer — train, render, and visualize results.

Pipeline stages (ablation levels):

  1. KS Raw — Karplus-Strong delay line, raw noise excitation, no filtering
  2. KS Filtered — KS + pitch-dependent excitation pre-filter + post-string LPF
  3. Full — KS Filtered + IIR resonator body (24-band)
  4. Target — ground-truth recording

Two training stages:

  • Stage 1 (50 epochs): MLP loop parameters + excitation gain
  • Stage 2 (50 epochs): Body resonator gains

0. Environment Setup¶

In [27]:
import os, sys, subprocess, time

# /workspace is the RunPod network volume root
# Structure: /workspace/data/, /workspace/results/, /workspace/guitar_poc_final.pth
# Scripts live in /workspace/AcousticGuitarDDSP/guitar-synth/
ROOT      = '/workspace'
SYNTH_DIR = os.path.join(ROOT, 'AcousticGuitarDDSP', 'guitar-synth')
DIFFKS_DIR = os.path.join(ROOT, 'diffKS_torchLPC')

DATA_DIR    = os.path.join(ROOT, 'data')
RESULTS_DIR = os.path.join(ROOT, 'results')
CHECKPOINT  = os.path.join(ROOT, 'guitar_poc_final.pth')

# Add diffKS library and guitar-synth to path
for p in [DIFFKS_DIR, SYNTH_DIR]:
    if p not in sys.path:
        sys.path.insert(0, p)

print(f'ROOT      : {ROOT}')
print(f'SYNTH_DIR : {SYNTH_DIR}  (exists: {os.path.isdir(SYNTH_DIR)})')
print(f'DIFFKS_DIR: {DIFFKS_DIR}  (exists: {os.path.isdir(DIFFKS_DIR)})')

import torch
if torch.cuda.is_available():
    print(f'\nGPU : {torch.cuda.get_device_name(0)}')
    print(f'VRAM: {torch.cuda.get_device_properties(0).total_memory / 1e9:.1f} GB')
else:
    print('\nNo CUDA GPU found — CPU will be used (training will be slow)')

# List training data
if os.path.isdir(DATA_DIR):
    wav_files = sorted(f for f in os.listdir(DATA_DIR) if f.lower().endswith('.wav'))
    print(f'\nData directory : {DATA_DIR}')
    print(f'WAV files found: {len(wav_files)}')
    for f in wav_files:
        print(f'  {f}')
else:
    print(f'\nWARNING: {DATA_DIR} not found')
    print('Place guitar WAV files named like guitar80Hz.wav, guitar220Hz.wav, etc.')

ROOT      : /workspace
SYNTH_DIR : /workspace/AcousticGuitarDDSP/guitar-synth  (exists: True)
DIFFKS_DIR: /workspace/diffKS_torchLPC  (exists: True)

GPU : NVIDIA RTX A4500
VRAM: 21.0 GB

Data directory : /workspace/data
WAV files found: 12
  guitar109Hz.wav
  guitar143Hz.wav
  guitar162Hz.wav
  guitar193Hz.wav
  guitar220Hz.wav
  guitar244Hz.wav
  guitar290Hz.wav
  guitar326Hz.wav
  guitar390Hz.wav
  guitar491Hz.wav
  guitar657Hz.wav
  guitar80Hz.wav

1. Train¶

Runs train.py — two-stage training (~5 min on A4000). Set SKIP_IF_EXISTS = True to skip training if a checkpoint already exists.

In [ ]:
SKIP_IF_EXISTS = True  # Set True to skip if checkpoint already exists, false to retrain from scratch

if SKIP_IF_EXISTS and os.path.exists(CHECKPOINT):
    print(f'Checkpoint found at {CHECKPOINT} — skipping training.')
    print('Set SKIP_IF_EXISTS = False to retrain from scratch.')
else:
    if os.path.exists(CHECKPOINT):
        print(f'Existing checkpoint will be overwritten: {CHECKPOINT}\n')

    print('Starting training...\n')
    t0 = time.time()

    proc = subprocess.Popen(
        [sys.executable, os.path.join(SYNTH_DIR, 'train.py')],
        cwd=ROOT,
        stdout=subprocess.PIPE,
        stderr=subprocess.STDOUT,
        text=True,
        bufsize=1
    )

    for line in proc.stdout:
        print(line, end='', flush=True)

    proc.wait()
    elapsed = time.time() - t0
    m, s = divmod(int(elapsed), 60)

    if proc.returncode == 0:
        print(f'\nTraining complete in {m}m {s:02d}s')
        print(f'Checkpoint saved: {CHECKPOINT}')
    else:
        print(f'\nTraining FAILED (exit code {proc.returncode})')

Checkpoint found at /workspace/guitar_poc_final.pth — skipping training.
Set SKIP_IF_EXISTS = False to retrain from scratch.

2. Render¶

Runs render.py — synthesizes low / mid / high training notes plus an interpolated 440 Hz note. Outputs WAV files and spectrogram PNGs to results/.

In [29]:
if not os.path.exists(CHECKPOINT):
    print(f'ERROR: Checkpoint not found at {CHECKPOINT}')
    print('Run the training cell first.')
else:
    print('Running render.py...\n')
    t0 = time.time()

    proc = subprocess.Popen(
        [sys.executable, os.path.join(SYNTH_DIR, 'render.py')],
        cwd=ROOT,
        stdout=subprocess.PIPE,
        stderr=subprocess.STDOUT,
        text=True,
        bufsize=1
    )

    for line in proc.stdout:
        print(line, end='', flush=True)

    proc.wait()
    elapsed = time.time() - t0

    if proc.returncode == 0:
        pngs = sorted(f for f in os.listdir(RESULTS_DIR) if f.endswith('.png'))
        print(f'\nRender complete in {elapsed:.1f}s')
        print(f'Output PNGs ({len(pngs)}):')
        for p in pngs:
            print(f'  {p}')
    else:
        print(f'\nRender FAILED (exit code {proc.returncode})')

Running render.py...

Using device: cuda
Model loaded from guitar_poc_final.pth
/opt/conda/lib/python3.11/site-packages/numba/cuda/dispatcher.py:536: NumbaPerformanceWarning: Grid size 8 will likely result in GPU under-utilization due to low occupancy.
  warn(NumbaPerformanceWarning(msg))
/opt/conda/lib/python3.11/site-packages/numba/cuda/dispatcher.py:536: NumbaPerformanceWarning: Grid size 8 will likely result in GPU under-utilization due to low occupancy.
  warn(NumbaPerformanceWarning(msg))

[low] 80.00 Hz  (gain_logit=1.805  mix_logit=-2.683  post_logit=1.561)
  Saved: results/low_80Hz_target.wav
  Saved: results/low_80Hz_target_spec.png
  Saved: results/low_80Hz_full.wav
  Saved: results/low_80Hz_full_spec.png
  Saved: results/low_80Hz_ks_filtered.wav
  Saved: results/low_80Hz_ks_filtered_spec.png
  Saved: results/low_80Hz_ks_raw.wav
  Saved: results/low_80Hz_ks_raw_spec.png
  Saved: results/low_80Hz_comparison.png

[mid] 244.00 Hz  (gain_logit=1.805  mix_logit=-2.683  post_logit=1.561)
  Saved: results/mid_244Hz_target.wav
  Saved: results/mid_244Hz_target_spec.png
  Saved: results/mid_244Hz_full.wav
  Saved: results/mid_244Hz_full_spec.png
  Saved: results/mid_244Hz_ks_filtered.wav
  Saved: results/mid_244Hz_ks_filtered_spec.png
  Saved: results/mid_244Hz_ks_raw.wav
  Saved: results/mid_244Hz_ks_raw_spec.png
  Saved: results/mid_244Hz_comparison.png

[high] 657.00 Hz  (gain_logit=1.805  mix_logit=-2.683  post_logit=1.561)
  Saved: results/high_657Hz_target.wav
  Saved: results/high_657Hz_target_spec.png
  Saved: results/high_657Hz_full.wav
  Saved: results/high_657Hz_full_spec.png
  Saved: results/high_657Hz_ks_filtered.wav
  Saved: results/high_657Hz_ks_filtered_spec.png
  Saved: results/high_657Hz_ks_raw.wav
  Saved: results/high_657Hz_ks_raw_spec.png
  Saved: results/high_657Hz_comparison.png

--- Trained Body Gains ---
  Band  1: -0.004711
  Band  2: 0.075090
  Band  3: 0.030749
  Band  4: 0.029494
  Band  5: 0.063821
  Band  6: 0.075157
  Band  7: 0.076651
  Band  8: 0.089054
  Band  9: 0.098708
  Band 10: 0.100579
  Band 11: 0.109596
  Band 12: 0.112923
  Band 13: 0.118436
  Band 14: 0.120916
  Band 15: 0.123861
  Band 16: 0.125368
  Band 17: 0.127543
  Band 18: 0.128404
  Band 19: 0.127792
  Band 20: 0.129089
  Band 21: 0.130536
  Band 22: 0.131804
  Band 23: 0.132941
  Band 24: 0.133476

--- Per-note MLP outputs ---
  [low] 80.0Hz | gain_logit=1.049 g=0.7405 | mix_logit=-0.889 p=0.291 | post_logit=0.106 cutoff_frac=0.527
  [mid] 244.0Hz | gain_logit=1.322 g=0.7896 | mix_logit=-2.574 p=0.071 | post_logit=0.633 cutoff_frac=0.653
  [high] 657.0Hz | gain_logit=2.012 g=0.8820 | mix_logit=-3.000 p=0.047 | post_logit=1.959 cutoff_frac=0.876

Saved interpolated note: results/interp_440Hz.wav

Render complete in 41.2s
Output PNGs (16):
  high_657Hz_comparison.png
  high_657Hz_full_spec.png
  high_657Hz_ks_filtered_spec.png
  high_657Hz_ks_raw_spec.png
  high_657Hz_target_spec.png
  interp_440Hz_spec.png
  low_80Hz_comparison.png
  low_80Hz_full_spec.png
  low_80Hz_ks_filtered_spec.png
  low_80Hz_ks_raw_spec.png
  low_80Hz_target_spec.png
  mid_244Hz_comparison.png
  mid_244Hz_full_spec.png
  mid_244Hz_ks_filtered_spec.png
  mid_244Hz_ks_raw_spec.png
  mid_244Hz_target_spec.png

3. Ablation Comparisons¶

4-panel spectrogram per note: Target | Full | KS Filtered | KS Raw

In [30]:
%matplotlib inline
import matplotlib.pyplot as plt
import matplotlib.image as mpimg
import base64
from IPython.display import display, HTML
import glob

def embed_audio(wav_path):
    """Render an <audio> tag with base64-encoded MP3 data URI.
    Works in live notebook AND in nbconvert HTML export (unlike IPython.display.Audio
    which relies on a JS widget that nbconvert strips).
    Falls back to WAV if ffmpeg is unavailable."""
    try:
        result = subprocess.run(
            ['ffmpeg', '-y', '-i', wav_path,
             '-acodec', 'libmp3lame', '-q:a', '2', '-f', 'mp3', 'pipe:1'],
            capture_output=True, timeout=30
        )
        if result.returncode == 0:
            b64 = base64.b64encode(result.stdout).decode()
            return HTML(f'<audio controls><source src="data:audio/mpeg;base64,{b64}" type="audio/mpeg"></audio>')
    except (FileNotFoundError, subprocess.TimeoutExpired):
        pass
    # Fallback: WAV data URI
    with open(wav_path, 'rb') as f:
        b64 = base64.b64encode(f.read()).decode()
    return HTML(f'<audio controls><source src="data:audio/wav;base64,{b64}" type="audio/wav"></audio>')

4. Individual Spectrograms: Target vs Full Synthesis¶

In [31]:
NOTES = ['low', 'mid', 'high']

# Discover Hz labels from filenames
all_pngs = sorted(glob.glob(os.path.join(RESULTS_DIR, '*.png')))
note_tags = []
for note in NOTES:
    matches = [os.path.basename(p) for p in all_pngs if os.path.basename(p).startswith(note + '_')]
    if matches:
        # Extract e.g. "low_80Hz" from "low_80Hz_target_spec.png"
        tag = '_'.join(matches[0].split('_')[:2])  # e.g. low_80Hz
        note_tags.append(tag)

if not note_tags:
    print('No result files found. Run the Render cell first.')
else:
    for tag in note_tags:
        target_path = os.path.join(RESULTS_DIR, f'{tag}_target_spec.png')
        full_path   = os.path.join(RESULTS_DIR, f'{tag}_full_spec.png')

        if not (os.path.exists(target_path) and os.path.exists(full_path)):
            continue

        fig, axes = plt.subplots(1, 2, figsize=(18, 4))
        for ax, path, label in zip(axes,
                                   [target_path, full_path],
                                   ['Target', 'Full Synthesis']):
            ax.imshow(mpimg.imread(path))
            ax.axis('off')
            ax.set_title(f'{label} — {tag.replace("_", " ")}', fontsize=11)
        fig.suptitle(tag.replace('_', ' '), fontsize=13, y=1.02)
        plt.tight_layout()
        plt.show()
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image

5. Per-Stage Ablation Spectrograms¶

In [32]:
STAGES = [
    ('target_spec',  'Target'),
    ('ks_raw_spec',  'KS Raw'),
    ('ks_filtered_spec', 'KS Filtered'),
    ('full_spec',    'Full (+ Body)'),
]

if not note_tags:
    print('No result files found. Run the Render cell first.')
else:
    for tag in note_tags:
        paths  = [os.path.join(RESULTS_DIR, f'{tag}_{suffix}.png') for suffix, _ in STAGES]
        labels = [label for _, label in STAGES]
        existing = [(p, l) for p, l in zip(paths, labels) if os.path.exists(p)]

        if not existing:
            continue

        n = len(existing)
        fig, axes = plt.subplots(1, n, figsize=(6 * n, 4))
        if n == 1:
            axes = [axes]
        for ax, (path, label) in zip(axes, existing):
            ax.imshow(mpimg.imread(path))
            ax.axis('off')
            ax.set_title(label, fontsize=10)
        fig.suptitle(f'Pipeline stages — {tag.replace("_", " ")}', fontsize=13, y=1.02)
        plt.tight_layout()
        plt.show()
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image

6. Interpolated Note (440 Hz)¶

In [33]:
interp_spec = glob.glob(os.path.join(RESULTS_DIR, 'interp_*_spec.png'))
interp_wav  = glob.glob(os.path.join(RESULTS_DIR, 'interp_*.wav'))

if not interp_spec:
    print('No interpolated spectrogram found. Run the Render cell first.')
else:
    img = mpimg.imread(interp_spec[0])
    fig, ax = plt.subplots(figsize=(12, 4))
    ax.imshow(img)
    ax.axis('off')
    ax.set_title('Interpolated synthesis — 440 Hz (unseen pitch)', fontsize=12)
    plt.tight_layout()
    plt.show()

if interp_wav:
    print('Interpolated audio:')
    display(embed_audio(interp_wav[0]))
No description has been provided for this image 
Interpolated audio:

7. Audio Playback — All Notes¶

In [34]:
AUDIO_VARIANTS = ['target', 'full', 'ks_filtered', 'ks_raw']

if not note_tags:
    print('No result files found. Run the Render cell first.')
else:
    for tag in note_tags:
        print(f'\n── {tag.replace("_", " ").upper()} ──')
        for variant in AUDIO_VARIANTS:
            wav = os.path.join(RESULTS_DIR, f'{tag}_{variant}.wav')
            if os.path.exists(wav):
                print(f'  {variant}:')
                display(embed_audio(wav))

── LOW 80HZ ──
  target:

  full:

  ks_filtered:

  ks_raw:

── MID 244HZ ──
  target:

  full:

  ks_filtered:

  ks_raw:

── HIGH 657HZ ──
  target:

  full:

  ks_filtered:

  ks_raw:

8. Export to Self-Contained HTML¶

Run the cell below after all other cells have been executed. The HTML file will have all spectrograms and audio players embedded as base64 — no external files needed.

In [ ]:
NOTEBOOK_PATH = os.path.join(SYNTH_DIR, 'notebook.ipynb')
HTML_PATH     = os.path.join(ROOT, 'notebook.html')

proc = subprocess.run(
    [
        sys.executable, '-m', 'nbconvert',
        '--to', 'html',
        '--no-input',
        '--HTMLExporter.sanitize_html=False',  # bleach strips <audio> + data: URIs by default
        '--output', HTML_PATH,
        NOTEBOOK_PATH,
    ],
    capture_output=True, text=True
)

if proc.returncode == 0:
    size_mb = os.path.getsize(HTML_PATH) / 1e6
    print(f'Exported: {HTML_PATH}  ({size_mb:.1f} MB)')
else:
    print('Export failed:')
    print(proc.stderr)