How to use ssh-agent from devcontainer CLI ?

I have a devcontainer that I will be using in vscode and in CI.

In the CI we use devcontainer up/exec

in vscode, we see there is a dev container helper that sets up ssh-agent.

In the CI we setup ssh-agent and add the key, but I cannot find a way to have the same devcontainer works in both environment.

I run vscode on windows and remote container are on linux. I can mount my SSH_AUTH_SOCK to /tmp/ssh-agent.socket and then I can start ssh-agent -a /tmp/ssh-agent.socket, but ssh-add -L still report no identities and outside of the container ssh-add -L do report at least one key that I did add to the ssh-agent.

Would you have an example of how one can use devcontainer with docker-compose to access the local ssh-agent, in a compatible way that would also work within vscode itself?

(when I load the same devcontainer in vscode, the SSH_AUTH_SOCK gets transformed into a ".", yet my linux remove is using ssh-agent and has a proper SSH_AUTH_SOCK value). Somehow I think the Windows version is being used, for which no SSH_AUTH_SOCK exists, but I do have the ssh-agent running on Windows.

I would definitely prefer that the remote container uses the remote linux ssh-agent than my windows.

Curious as well, there seems to be tons of old documentation on how to do this (https://stackoverflow.com/questions/43418188/ssh-agent-forwarding-during-docker-build) but it appears there is more modern (still old) tooling with docker build --ssh too. moby/moby#6396 (comment)

The ssh-agent forwarding is part of the Dev Containers extension and not part of the Dev Containers CLI. You could mount the ssh-agent's socket and then point SSH_AUTH_SOCK at it.

Strangely when I did mount the ssh-auth-socket to let's say /tmp/ssh-agent.socket and then set SSH_AUTH_SOCK to /tmp/ssh-agent.socket, I could confirm the mount was done and the env was set, but ssh-add was still unable to talk to the ssh-agent itself.

I then start a ssh-agent within the devcontainer itself telling it to use /tmp/ssh-agent.socket and ssh-add was happy but still not finding any of my key which I had in my how ssh-agent.

The other issue, is that when I did so, within vscode things stop working, because the SSH_AUTH_SOCK got converted to "." instead of the real SSH_AUTH_SOCK value, even thougth the dev conatiners extension is able to find that value propery.

    "runArgs": [
       "-e",
       "SSH_AUTH_SOCK=/tmp/ssh-agent.socket",
       "-v",
       "${env:SSH_AUTH_SOCK}:/tmp/ssh-agent.socket"
    ],

and docker-compose.yml:

    volumes:
    - ..:/workspaces:cached
    - ${SSH_AUTH_SOCK}:/tmp/ssh-agent.socket

If I do that in a docker container it works, but when I do it via the devcontainer cli it doesn't work.

Anyway it would be really nice for devcontainer teams to align and provide an example on how to use ssh-agent in devcontainer CLI in a way compatible with VSCode that is compatible with all OS/remote or not, etc. as it doesn't seem that easy to get working.

You only need to run an ssh-agent on your local machine (where the private keys are), not inside the container. Note that "runArgs" doesn't apply to Docker Compose. I can't get it to work on Mac right now, maybe this has changed.

Another option would be to mount ~/.ssh in the container to get access to the private keys.

Christof Marti (@chrmarti)

Christof Marti (@chrmarti) I would prefer the option of ssh-agent because getting access to the key doesn't get you access to the passphrase to unlock the key which then raises another challenge.

It also simplifies cases where the key to use may not be in the .ssh, some users prefer to use a different folder to make it harder to attackers to find those, etc.

when I do not start the ssh-agent within the container, even thought the socket is mounted, ssh-add simply fail to connect to it and report an error.

I am using VSCode on Windows, then connect to Linux via Remote-SSH and use devcontainer.

Using VSCode, the dev containers helper connect my "Windows" ssh-agent to my devcontainer... which in itself is strange, as I would really like my remote linux to be the ssh-agent, as to not share my keys on my windows machine like I have to now. But that is another matter.

And as indicated then using SSH_AUTH_SOCK env variable it breaks VSCode because then SSH_AUTH_SOCK becomes . and the docker-compose complains that

.:/tmp/ssh-agent.socket

is not a valid volume format.

It somehow look like I would need to start VSCode with SSH_AUTH_SOCK set to my linux SSH_AUTH_SOCK so that VSCode continue to work... which doesn't appear usable.

I agree, using the ssh-agent seems preferable.

Using VSCode, the dev containers helper connect my "Windows" ssh-agent to my devcontainer... which in itself is strange, as I would really like my remote linux to be the ssh-agent, as to not share my keys on my windows machine like I have to now. But that is another matter.

That should actually work. If SSH_AUTH_SOCK is properly set in your remote machine's shell startup scripts, the Dev Containers extension should prefer that over the local one. Try running with Dev Container's log level set to 'trace' in the user settings to get the env variables logged as seen by the extension.

And as indicated then using SSH_AUTH_SOCK env variable it breaks VSCode because then SSH_AUTH_SOCK becomes . and the docker-compose complains that

.:/tmp/ssh-agent.socket

is not a valid volume format.

Docker Compose should do the variable replacement there. Maybe another indicator that SSH_AUTH_SOCK isn't properly picked up on the remote machine.

That should actually work. If SSH_AUTH_SOCK is properly set in your remote machine's shell startup scripts, the Dev Containers extension should prefer that over the local one. Try running with Dev Container's log level set to 'trace' in the user settings to get the env variables logged as seen by the extension.

SSH_AUTH_SOCK is set by my .bashrc but in my linux, it works nicely when I use it with ssh, but vscode still translate to .

One interesting thing is that my ssh-agent in order to use a single socket start ssh-agent, create a symlink to the socket and then set SSH_AUTH_SOCK to the symlink.

if [ ! -S /tmp/yk-ssh_auth_sock ]; then
    eval $(ssh-agent)
    ln -sf "$SSH_AUTH_SOCK" /tmp/yk-ssh_auth_sock
fi
export SSH_AUTH_SOCK=/tmp/yk-ssh_auth_sock

So maybe that could be a reason for this issue. Another could be that .bashrc is only used for some interactive login and I may need to put it in a .profile or .bash_profile, I will try that to see if things change.

But I can assure you that it currently prefer my Windows ssh-agent over my Linux one.

[11009 ms] Start: Launching Dev Containers helper.
[11010 ms] ssh-agent: SSH_AUTH_SOCK in container (/tmp/vscode-ssh-auth-f0cca852-df04-4f31-8740-5fdba9bd0967.sock) forwarded to local host (\\.\pipe\openssh-ssh-agent).
[11010 ms] X11 forwarding: DISPLAY not set on host.

I wish this part was a little more verbose and somehow provide clues as to why my Linux ssh-agent was not found, maybe some kind of warning that the SSH_AUTH_SOCK is not set, the extension fallback to the native host ssh-agent ? With some URL pointing to how to properly set SSH_AUTH_SOCK in a way for Remote-SSH to detect it.

Because the issue my SSH_AUTH_SOCK is set when I used ssh manually, it is also set when I used remote-ssh feature of VSCode, it is only apparently not set when I used the remote-devcontainer plugin feature.

I have removed the symlink by using ssh-agent -a and no change.

if [ ! -S /tmp/yk-ssh_auth_sock ]; then
    eval $(ssh-agent -a /tmp/yk-ssh_auth_sock)
fi

still attach to my windows ssh-agent:

[9765 ms] Start: Launching Dev Containers helper.
[9765 ms] ssh-agent: SSH_AUTH_SOCK in container (/tmp/vscode-ssh-auth-1c2a37c7-9284-4ad4-99a7-7013dc50c28a.sock) forwarded to local host (\\.\pipe\openssh-ssh-agent).
[9766 ms] X11 forwarding: DISPLAY not set on host.

I also added the ssh-agent setup in my .bash_profile and .profile, no impact.

Could you run with the Dev Container's log level set to 'trace' in the user settings to get the env variables logged as seen by the extension?

I will also improve logging for this. For now we can use the trace logging output.

[18871 ms] Start: Run: docker inspect --type container 5bbf06ee9b61785eb5ccd1e58e556086a53a497e9054160c6248d01a9906f1d7
[18955 ms] Stop (84 ms): Run: docker inspect --type container 5bbf06ee9b61785eb5ccd1e58e556086a53a497e9054160c6248d01a9906f1d7
[18956 ms] Stop (85 ms): Inspecting container
[18959 ms] Start: Run in container: /bin/sh
[18973 ms] Start: Run in container: uname -m
[19145 ms] x86_64
[19145 ms] 
[19145 ms] Stop (172 ms): Run in container: uname -m
[19145 ms] Start: Run in container: (cat /etc/os-release || cat /usr/lib/os-release) 2>/dev/null
[19153 ms] NAME="Ubuntu"
VERSION="20.04.6 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.6 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal
[19153 ms] 
[19153 ms] Stop (8 ms): Run in container: (cat /etc/os-release || cat /usr/lib/os-release) 2>/dev/null
[19154 ms] Start: Run in container: cat /etc/passwd
[19157 ms] Stop (3 ms): Run in container: cat /etc/passwd
[19159 ms] Start: Run in container: test -f '/var/devcontainer/.patchEtcEnvironmentMarker'
[19161 ms] 
[19162 ms] 
[19162 ms] Exit code 1
[19162 ms] Stop (3 ms): Run in container: test -f '/var/devcontainer/.patchEtcEnvironmentMarker'
[19163 ms] Start: Run in container: /bin/sh
[19176 ms] Start: Run in container: test ! -f '/var/devcontainer/.patchEtcEnvironmentMarker' && set -o noclobber && mkdir -p '/var/devcontainer' && { > '/var/devcontainer/.patchEtcEnvironmentMarker' ; } 2> /dev/null
[19357 ms] 
[19357 ms] 
[19357 ms] Stop (181 ms): Run in container: test ! -f '/var/devcontainer/.patchEtcEnvironmentMarker' && set -o noclobber && mkdir -p '/var/devcontainer' && { > '/var/devcontainer/.patchEtcEnvironmentMarker' ; } 2> /dev/null
[19357 ms] Start: Run in container: cat >> /etc/environment <<'etcEnvrionmentEOF'
[19365 ms] 
[19365 ms] 
[19365 ms] Stop (8 ms): Run in container: cat >> /etc/environment <<'etcEnvrionmentEOF'
[19366 ms] Start: Run in container: test -f '/var/devcontainer/.patchEtcProfileMarker'
[19368 ms] 
[19368 ms] 
[19368 ms] Exit code 1
[19368 ms] Stop (2 ms): Run in container: test -f '/var/devcontainer/.patchEtcProfileMarker'
[19369 ms] Start: Run in container: test ! -f '/var/devcontainer/.patchEtcProfileMarker' && set -o noclobber && mkdir -p '/var/devcontainer' && { > '/var/devcontainer/.patchEtcProfileMarker' ; } 2> /dev/null
[19375 ms] 
[19375 ms] 
[19375 ms] Stop (6 ms): Run in container: test ! -f '/var/devcontainer/.patchEtcProfileMarker' && set -o noclobber && mkdir -p '/var/devcontainer' && { > '/var/devcontainer/.patchEtcProfileMarker' ; } 2> /dev/null
[19376 ms] Start: Run in container: sed -i -E 's/((^|\s)PATH=)([^\$]*)$/\1${PATH:-\3}/g' /etc/profile || true
[19387 ms] 
[19387 ms] 
[19387 ms] Stop (11 ms): Run in container: sed -i -E 's/((^|\s)PATH=)([^\$]*)$/\1${PATH:-\3}/g' /etc/profile || true
[19389 ms] Stop (14350 ms): Resolving Remote
[19422 ms] Stop (15424 ms): Run in Host: /net/spindles/ykoehler/.vscode-remote-containers/bin/ee2b180d582a7f601fa6ecfdad8d9fd269ab1884/node /net/spindles/ykoehler/.vscode-remote-containers/dist/dev-containers-cli-0.286.0/dist/spec-node/devContainersSpecCLI.js up --container-session-data-folder /tmp/devcontainers-933f80f4-6726-4784-aa66-128ff75f113e1679597210270 --workspace-folder /work/krypton/dal --workspace-mount-consistency cached --id-label devcontainer.local_folder=/work/krypton/dal --id-label devcontainer.config_file=/work/krypton/dal/.devcontainer/devcontainer.json --log-level trace --log-format json --config /work/krypton/dal/.devcontainer/devcontainer.json --default-user-env-probe loginInteractiveShell --remove-existing-container --mount type=volume,source=vscode,target=/vscode,external=true --skip-post-create --update-remote-user-uid-default on --mount-workspace-git-root true --terminal-columns 298 --terminal-rows 37
[19425 ms] Start: Run in Host: docker inspect --type container 5bbf06ee9b61785eb5ccd1e58e556086a53a497e9054160c6248d01a9906f1d7
[19630 ms] Stop (205 ms): Run in Host: docker inspect --type container 5bbf06ee9b61785eb5ccd1e58e556086a53a497e9054160c6248d01a9906f1d7
[19631 ms] Start: Run in Host: docker exec -i -u root 5bbf06ee9b61785eb5ccd1e58e556086a53a497e9054160c6248d01a9906f1d7 /bin/sh -c echo "New container started. Keep-alive process started." ; export VSCODE_REMOTE_CONTAINERS_SESSION=933f80f4-6726-4784-aa66-128ff75f113e1679597210270 ; /bin/sh
[19836 ms] Start: Run in Host: /net/spindles/ykoehler/.vscode-remote-containers/bin/ee2b180d582a7f601fa6ecfdad8d9fd269ab1884/node /net/spindles/ykoehler/.vscode-remote-containers/dist/dev-containers-cli-0.286.0/dist/spec-node/devContainersSpecCLI.js read-configuration --workspace-folder /work/krypton/dal --id-label devcontainer.local_folder=/work/krypton/dal --id-label devcontainer.config_file=/work/krypton/dal/.devcontainer/devcontainer.json --container-id 5bbf06ee9b61785eb5ccd1e58e556086a53a497e9054160c6248d01a9906f1d7 --log-level trace --log-format json --config /work/krypton/dal/.devcontainer/devcontainer.json --mount-workspace-git-root true
[19899 ms] New container started. Keep-alive process started.
[20376 ms] @devcontainers/cli 0.35.0. Node.js v16.14.2. linux 5.15.0-53-generic x64.
[20375 ms] Start: Run: docker inspect --type container 5bbf06ee9b61785eb5ccd1e58e556086a53a497e9054160c6248d01a9906f1d7
[20463 ms] Stop (88 ms): Run: docker inspect --type container 5bbf06ee9b61785eb5ccd1e58e556086a53a497e9054160c6248d01a9906f1d7
[20496 ms] Stop (660 ms): Run in Host: /net/spindles/ykoehler/.vscode-remote-containers/bin/ee2b180d582a7f601fa6ecfdad8d9fd269ab1884/node /net/spindles/ykoehler/.vscode-remote-containers/dist/dev-containers-cli-0.286.0/dist/spec-node/devContainersSpecCLI.js read-configuration --workspace-folder /work/krypton/dal --id-label devcontainer.local_folder=/work/krypton/dal --id-label devcontainer.config_file=/work/krypton/dal/.devcontainer/devcontainer.json --container-id 5bbf06ee9b61785eb5ccd1e58e556086a53a497e9054160c6248d01a9906f1d7 --log-level trace --log-format json --config /work/krypton/dal/.devcontainer/devcontainer.json --mount-workspace-git-root true
[20499 ms] Start: Run in Host: docker-compose version --short
[21049 ms] Stop (550 ms): Run in Host: docker-compose version --short
[21448 ms] Start: Run in Host: /net/spindles/ykoehler/.vscode-remote-containers/bin/ee2b180d582a7f601fa6ecfdad8d9fd269ab1884/node /net/spindles/ykoehler/.vscode-remote-containers/dist/dev-containers-cli-0.286.0/dist/spec-node/devContainersSpecCLI.js read-configuration --workspace-folder /work/krypton/dal --id-label devcontainer.local_folder=/work/krypton/dal --id-label devcontainer.config_file=/work/krypton/dal/.devcontainer/devcontainer.json --container-id 5bbf06ee9b61785eb5ccd1e58e556086a53a497e9054160c6248d01a9906f1d7 --log-level trace --log-format json --config /work/krypton/dal/.devcontainer/devcontainer.json --include-merged-configuration --mount-workspace-git-root true
[21893 ms] @devcontainers/cli 0.35.0. Node.js v16.14.2. linux 5.15.0-53-generic x64.
[21893 ms] Start: Run: docker inspect --type container 5bbf06ee9b61785eb5ccd1e58e556086a53a497e9054160c6248d01a9906f1d7
[21979 ms] Stop (86 ms): Run: docker inspect --type container 5bbf06ee9b61785eb5ccd1e58e556086a53a497e9054160c6248d01a9906f1d7
[22013 ms] Stop (565 ms): Run in Host: /net/spindles/ykoehler/.vscode-remote-containers/bin/ee2b180d582a7f601fa6ecfdad8d9fd269ab1884/node /net/spindles/ykoehler/.vscode-remote-containers/dist/dev-containers-cli-0.286.0/dist/spec-node/devContainersSpecCLI.js read-configuration --workspace-folder /work/krypton/dal --id-label devcontainer.local_folder=/work/krypton/dal --id-label devcontainer.config_file=/work/krypton/dal/.devcontainer/devcontainer.json --container-id 5bbf06ee9b61785eb5ccd1e58e556086a53a497e9054160c6248d01a9906f1d7 --log-level trace --log-format json --config /work/krypton/dal/.devcontainer/devcontainer.json --include-merged-configuration --mount-workspace-git-root true
[22015 ms] Start: Inspecting container
[22016 ms] Start: Run in Host: docker inspect --type container 5bbf06ee9b61785eb5ccd1e58e556086a53a497e9054160c6248d01a9906f1d7
[22213 ms] Stop (197 ms): Run in Host: docker inspect --type container 5bbf06ee9b61785eb5ccd1e58e556086a53a497e9054160c6248d01a9906f1d7
[22214 ms] Stop (199 ms): Inspecting container
[22222 ms] Start: Run in Host: docker exec -i -u aos-dev -e VSCODE_REMOTE_CONTAINERS_SESSION=933f80f4-6726-4784-aa66-128ff75f113e1679597210270 5bbf06ee9b61785eb5ccd1e58e556086a53a497e9054160c6248d01a9906f1d7 /bin/sh
[22326 ms] Start: Run in container: uname -m
[22509 ms] x86_64
[22510 ms] 
[22510 ms] Stop (184 ms): Run in container: uname -m
[22510 ms] Start: Run in container: (cat /etc/os-release || cat /usr/lib/os-release) 2>/dev/null
[22609 ms] NAME="Ubuntu"
VERSION="20.04.6 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.6 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal
[22610 ms] 
[22610 ms] Stop (100 ms): Run in container: (cat /etc/os-release || cat /usr/lib/os-release) 2>/dev/null
[22611 ms] Start: Run in container: cat /etc/passwd
[22708 ms] Stop (97 ms): Run in container: cat /etc/passwd
[22709 ms] Start: Updating configuration state
[22909 ms] Stop (200 ms): Updating configuration state
[22910 ms] Start: Setup shutdown monitor
[22912 ms] Forking shutdown monitor: c:\Users\koehlery\.vscode\extensions\ms-vscode-remote.remote-containers-0.286.0\dist\shutdown\shutdownMonitorProcess \\.\pipe\vscode-remote-containers-188be55c-b66a-4687-984e-451d68846e1d-sock dockerCompose Trace c:\Users\koehlery\AppData\Roaming\Code\logs\20230323T124001\window1\exthost\ms-vscode-remote.remote-containers 1679597211250
[22928 ms] Stop (18 ms): Setup shutdown monitor
[22930 ms] Start: Run in container: test -d /home/aos-dev/.vscode-server
[23026 ms] 
[23027 ms] 
[23027 ms] Exit code 1
[23027 ms] Stop (97 ms): Run in container: test -d /home/aos-dev/.vscode-server
[23028 ms] Start: Run in container: test -d /home/aos-dev/.vscode-remote
[23127 ms] 
[23127 ms] 
[23128 ms] Exit code 1
[23128 ms] Stop (100 ms): Run in container: test -d /home/aos-dev/.vscode-remote
[23129 ms] Start: Run in container: test ! -f '/home/aos-dev/.vscode-server/data/Machine/.writeMachineSettingsMarker' && set -o noclobber && mkdir -p '/home/aos-dev/.vscode-server/data/Machine' && { > '/home/aos-dev/.vscode-server/data/Machine/.writeMachineSettingsMarker' ; } 2> /dev/null
[23156 ms] Starting monitor process...
[23180 ms] Received message: [{"type":"configure","sequence":0,"options":{"cwd":"/work/krypton/dal","dockerCLI":"docker","dockerComposeCLI":{"version":"1.29.2","cmd":"docker-compose","args":[]},"env":{"USER":"ykoehler","SSH_CLIENT":"15.111.190.46 59187 22","XDG_SESSION_TYPE":"tty","SHLVL":"0","MOTD_SHOWN":"pam","HOME":"/net/spindles/ykoehler","DBUS_SESSION_BUS_ADDRESS":"unix:path=/run/user/11846/bus","LOGNAME":"ykoehler","REMOTE_CONTAINERS_IPC":"","_":"dc1b38b5-793b-473c-9beb-1443b3852f14","XDG_SESSION_CLASS":"user","XDG_SESSION_ID":"24863","PATH":"/net/spindles/ykoehler/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin","XDG_RUNTIME_DIR":"/run/user/11846","REMOTE_CONTAINERS_SOCKETS":"[]","LANG":"en_US.UTF-8","SHELL":"/bin/bash","SSH_CONNECTION":"15.111.190.46 59187 10.82.21.252 22","XDG_DATA_DIRS":"/usr/share/gnome:/usr/local/share/:/usr/share/","DISPLAY":"1","ELECTRON_RUN_AS_NODE":"1","SSH_ASKPASS":"c:\\Users\\koehlery\\.vscode\\extensions\\ms-vscode-remote.remote-containers-0.286.0\\scripts\\ssh-askpass.bat","SSH_ASKPASS_REQUIRE":"force","VSCODE_SSH_ASKPASS_NODE":"C:\\Users\\koehlery\\AppData\\Local\\Programs\\Microsoft VS Code\\Code.exe","VSCODE_SSH_ASKPASS_EXTRA_ARGS":"--ms-enable-electron-run-as-node","VSCODE_SSH_ASKPASS_MAIN":"c:\\Users\\koehlery\\.vscode\\extensions\\ms-vscode-remote.remote-containers-0.286.0\\dist\\common\\sshAskpass.js","VSCODE_SSH_ASKPASS_HANDLE":"\\\\.\\pipe\\ssh-askpass-7e8dfe39-ed35-48e1-be65-950df360332b-sock","DOCKER_CONTEXT":"default","VSCODE_DOCKER_HOST":"ssh://slrubdev-ykoehler22"},"containerId":"5bbf06ee9b61785eb5ccd1e58e556086a53a497e9054160c6248d01a9906f1d7","user":"aos-dev","sessionId":"933f80f4-6726-4784-aa66-128ff75f113e1679597210270","shutdownAction":"stopCompose","projectName":"dal_devcontainer","composeFiles":["/work/krypton/dal/.devcontainer/docker-compose.yml"],"service":"krypton-c-env","logLevel":1,"delay":10000}}]
[23236 ms] 
[23236 ms] 
[23236 ms] Stop (107 ms): Run in container: test ! -f '/home/aos-dev/.vscode-server/data/Machine/.writeMachineSettingsMarker' && set -o noclobber && mkdir -p '/home/aos-dev/.vscode-server/data/Machine' && { > '/home/aos-dev/.vscode-server/data/Machine/.writeMachineSettingsMarker' ; } 2> /dev/null
[23237 ms] Start: Run in container: cat /home/aos-dev/.vscode-server/data/Machine/settings.json
[23343 ms] 
[23344 ms] cat: /home/aos-dev/.vscode-server/data/Machine/settings.json: No such file or directory
[23344 ms] Exit code 1
[23344 ms] Stop (107 ms): Run in container: cat /home/aos-dev/.vscode-server/data/Machine/settings.json
[23345 ms] Start: Run in container: test -d /home/aos-dev/.vscode-server/bin/ee2b180d582a7f601fa6ecfdad8d9fd269ab1884
[23453 ms] 
[23453 ms] 
[23453 ms] Exit code 1
[23453 ms] Stop (108 ms): Run in container: test -d /home/aos-dev/.vscode-server/bin/ee2b180d582a7f601fa6ecfdad8d9fd269ab1884
[23453 ms] Start: Run in container: test -d /vscode/vscode-server/bin/linux-x64/ee2b180d582a7f601fa6ecfdad8d9fd269ab1884
[23554 ms] 
[23554 ms] 
[23554 ms] Stop (101 ms): Run in container: test -d /vscode/vscode-server/bin/linux-x64/ee2b180d582a7f601fa6ecfdad8d9fd269ab1884
[23554 ms] Start: Run in container: mkdir -p '/home/aos-dev/.vscode-server/bin' && ln -snf '/vscode/vscode-server/bin/linux-x64/ee2b180d582a7f601fa6ecfdad8d9fd269ab1884' '/home/aos-dev/.vscode-server/bin/ee2b180d582a7f601fa6ecfdad8d9fd269ab1884'
[23661 ms] 
[23662 ms] 
[23662 ms] Stop (108 ms): Run in container: mkdir -p '/home/aos-dev/.vscode-server/bin' && ln -snf '/vscode/vscode-server/bin/linux-x64/ee2b180d582a7f601fa6ecfdad8d9fd269ab1884' '/home/aos-dev/.vscode-server/bin/ee2b180d582a7f601fa6ecfdad8d9fd269ab1884'
[23663 ms] Start: Run in Host: docker exec -i -u root 5bbf06ee9b61785eb5ccd1e58e556086a53a497e9054160c6248d01a9906f1d7 /bin/sh
[23664 ms] Start: Launching Dev Containers helper.
[23665 ms] ssh-agent: SSH_AUTH_SOCK in container (/tmp/vscode-ssh-auth-542f4882-bd97-4ec9-84e9-5890f4f5d07b.sock) forwarded to local host (\\.\pipe\openssh-ssh-agent).
[23666 ms] X11 forwarding: DISPLAY not set on host.
[23666 ms] Start: Run in Host: gpgconf --list-dir agent-extra-socket
[23776 ms] Start: Run in container: touch '/vscode/vscode-server/bin/linux-x64/ee2b180d582a7f601fa6ecfdad8d9fd269ab1884'
[23879 ms] Stop (213 ms): Run in Host: gpgconf --list-dir agent-extra-socket
[23880 ms] /run/user/11846/gnupg/S.gpg-agent.extra
[23881 ms] 
[23881 ms] Start: Run in container: gpgconf --list-dir agent-socket
[23942 ms] 
[23942 ms] 
[23942 ms] Stop (166 ms): Run in container: touch '/vscode/vscode-server/bin/linux-x64/ee2b180d582a7f601fa6ecfdad8d9fd269ab1884'
[23993 ms] /home/aos-dev/.gnupg/S.gpg-agent
[23993 ms] 
[23993 ms] Stop (112 ms): Run in container: gpgconf --list-dir agent-socket
[23993 ms] Start: Run in container: gpgconf --list-dir homedir
[24101 ms] /home/aos-dev/.gnupg
[24102 ms] 
[24102 ms] Stop (109 ms): Run in container: gpgconf --list-dir homedir
[24102 ms] Start: Run in container: ls '/home/aos-dev/.gnupg/private-keys-v1.d' 2>/dev/null
[24204 ms] 
[24205 ms] 
[24205 ms] Exit code 2
[24205 ms] Stop (103 ms): Run in container: ls '/home/aos-dev/.gnupg/private-keys-v1.d' 2>/dev/null
[24205 ms] Start: Run in container: mkdir -p -m 700 '/home/aos-dev/.gnupg'
[24302 ms] 
[24302 ms] 
[24302 ms] Stop (97 ms): Run in container: mkdir -p -m 700 '/home/aos-dev/.gnupg'
[24302 ms] Start: Run in container: command -v docker >/dev/null 2>&1
[24302 ms] Start: Run in Host: gpgconf --list-dir homedir
[24403 ms] 
[24403 ms] 
[24403 ms] Stop (101 ms): Run in container: command -v docker >/dev/null 2>&1
[24404 ms] Start: Run in Host: docker exec -i -u aos-dev 5bbf06ee9b61785eb5ccd1e58e556086a53a497e9054160c6248d01a9906f1d7 /bin/sh
[24404 ms] Stop (740 ms): Launching Dev Containers helper.
[24404 ms] userEnvProbe: loginInteractiveShell (default)
[24405 ms] Start: Run in container: test -f '/tmp/devcontainers-933f80f4-6726-4784-aa66-128ff75f113e1679597210270/env-loginInteractiveShell.json'
[24512 ms] Stop (210 ms): Run in Host: gpgconf --list-dir homedir
[24514 ms] /net/spindles/ykoehler/.gnupg
[24514 ms] 
[24515 ms] 
[24516 ms] 
[24516 ms] Exit code 1
[24516 ms] Stop (111 ms): Run in container: test -f '/tmp/devcontainers-933f80f4-6726-4784-aa66-128ff75f113e1679597210270/env-loginInteractiveShell.json'
[24517 ms] Start: Run in container: echo ~
[24517 ms] Start: Run in container: gpgconf --list-dir homedir
[24517 ms] userEnvProbe: not found in cache
[24518 ms] userEnvProbe shell: /bin/bash
[24614 ms] /home/aos-dev/.gnupg
[24615 ms] 
[24615 ms] Stop (98 ms): Run in container: gpgconf --list-dir homedir
[24615 ms] Start: Run in container: # Test for /home/aos-dev/.ssh/known_hosts and ssh
[24702 ms] /home/aos-dev
[24703 ms] 
[24704 ms] Stop (187 ms): Run in container: echo ~
[24704 ms] Start: Run in container: cat <<'EOF-/tmp/vscode-remote-containers-542f4882-bd97-4ec9-84e9-5890f4f5d07b.js' >/tmp/vscode-remote-containers-542f4882-bd97-4ec9-84e9-5890f4f5d07b.js
[24726 ms] 
[24727 ms]

In the area around ssh-agent, there apparently are no trace logs.

That doesn't seem to be the full log. Check for userEnvProbe in the full log, that will run twice, first on the SSH server and then in the container. For the ssh-agent forwarding on the SSH server, the first is of interest.

Also check your local ~/.ssh/config for ForwardAgent yes, that might overwrite SSH_AUTH_SOCK.

You can find log files from previous sessions with F1 > Dev Containers Developer: Show All Logs....

Would it be possible to add a cli arg to make this more easily possible? I am confused how to do this.

Ryan Butler (@TheButlah)

Ryan Butler (@TheButlah) The ssh-agent is only forwarded when using the Dev Containers extension for VS Code. The Dev Containers CLI does not do that yet. Labelling as feature request.

One wishes I have is that once devcontainer/cli does it, that vscode stop doing it on its own and relies on the devcontainer/cli tool to do it instead. The idea is that there is a single mechanism to understand/document/debug.

It also seems like there is a similar system related to GPG, likely both would need to be moved from vscode to devcontainer/cli.

BTW, to solve this, I had to both mount the agent socket into the container, set the permission of the socket to allow the user to access it, and set SSH_AUTH_SOCK env var to point to the socket.

On mac, the socket path was the magic path of /run/host-services/ssh-auth.sock, whereas on linux this is likely dynamic.

I still advocate that this functionality be implemented directly in the devcontainers cli, as figuring out how to do all of that was a nightmare.

Ryan Butler (@TheButlah)

Ryan Butler (@TheButlah) I will see if I can repro your result, I am on Ubuntu and I have attempted many things, without success so far. Also there seem to be 2 modes of usage in devcontainer, either Dockefile, docker-compose. I am using docker-compose, which may be slightly different sometimes. I remember having tried to mount the socket in docker-compose and check permission, also move the ssh-agent socket location from ~/.ssh/ which was not accessible to /tmp/, and then also passing an SSH_AUTH_SOCK value that made sense within the container, but where I had success with normal docker container, it didn't work, from memory, using devcontainer.

JFYI by mounting /run/host-services/ssh-auth.sock on your devcontainer.json like:

	"mounts": [
		"source=/run/host-services/ssh-auth.sock,target=/run/host-services/ssh-auth.sock,type=bind"
	],

Is unfortunately non-portable to other OS like Windows. Native support from devcontainer/cli is required.

I would also like to flag that vscode is additionally do some more magic that involved even .gitconfig. I wonder if is there a plan to migrate all these essential functionality to devcontainer/cli?

Based on what I'm gathering (and for my several years of trying to do this lazily across docker and k8s) this will require a wrapper so you can mux PTY/std fds and agent sock to target container over stdin/stdout. Which I'm guessing is what the extension does when it runs, based on what I see in lsof. Some similar logic to adapt to whatever Windows needs. Frankly, the easiest solution right now is to run an sshd inside the container (that's super ugly, but will result in less complaints from my devs that writing my own minimally used thing) since it will handle all this stuff out of the box.

It's a nontrivial effort because basically the same pty logic from docker's run/exec commands to setup the terminal and do the right thing over stdin/stdout to the daemon. Then, each channel (pty/ssh agent/gpg agent/x11 socket/...) needs its own flow control to avoid head-of-line blocking before demux. BTW, this technique is definitely required in k8s where you can't easily just reach in and mount stuff. Of course if Microsoft didn't hold the dev containers extension behind the proprietary firewall, this could probably be made to work a bit quicker with some copy+paest engineering.

My use-case is simple: I want to develop in containers without vscode or similar tooling and simply hop in, vim it good, commit, and push all from within the container. I should be able to leave the containers running, drop my ssh session, hop back in, deal with it being run as another user (so realistically, the socket should be proxied inside the container) and generally be as clean every time i do a devcontainer exec.

If the issue is socket permissions (e.g. host user is root, container user is non-root), you can use socat in the container to proxy to the host's ssh-agent socket. Stick something like this in the postCreateCommand lifecycle step:

# usage:
# devcontainer up \
#   --remote-env "SSH_AUTH_SOCK=/tmp/ssh-auth-sock" \
#   --mount "type=bind,source=${SSH_AUTH_SOCK},target=/tmp/ssh-auth-sock" \
#   ... etc.

# If SSH_AUTH_SOCK exists and isn't owned by the container user, use socat to proxy
# between a socket owned by the container user and the one mounted in from the host
if test -n "${SSH_AUTH_SOCK:-}" \
&& test "${SSH_AUTH_SOCK}" != ~/.ssh/socket \
&& test "$(stat -c "%u:%g" "${SSH_AUTH_SOCK}")" != "$(id -u):$(id -g)"; then
    ssh_auth_sock=~/.ssh/socket;
    # If the remapped ssh socket doesn't already exist
    if ! test -f "${ssh_auth_sock}"; then
        # Map the host's ssh-agent socket to one owned by the container user
        sudo --background socat "UNIX-LISTEN:${ssh_auth_sock},fork,user=$(id -nu),group=$(id -ng),mode=777" "UNIX-CONNECT:${SSH_AUTH_SOCK}";
        sleep 5;
    fi
    # test the remapped ssh-agent socket
    SSH_AUTH_SOCK="${ssh_auth_sock}" ssh-add -L 1>&2;
    # set SSH_AUTH_SOCK to the new remapped one
    echo "SSH_AUTH_SOCK=${ssh_auth_sock}" >> ~/.bashrc;
    export SSH_AUTH_SOCK="${ssh_auth_sock}";
    unset ssh_auth_sock;
fi

Explicitly set SSH_AUTH_SOCK for devcontainer CLI

ci: sign devcontainer commits through the forwarded ssh agent (#119)

Navigation Menu

Repository navigation

How to use ssh-agent from devcontainer CLI ? #441

2 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

How to use ssh-agent from devcontainer CLI ? #441

Description

Activity

metaskills commented on Mar 16, 2023

chrmarti commented on Mar 21, 2023

hpe-ykoehler commented on Mar 21, 2023

hpe-ykoehler commented on Mar 21, 2023

chrmarti commented on Mar 21, 2023

hpe-ykoehler commented on Mar 21, 2023

chrmarti commented on Mar 22, 2023

hpe-ykoehler commented on Mar 22, 2023

hpe-ykoehler commented on Mar 22, 2023

chrmarti commented on Mar 23, 2023

chrmarti commented on Mar 23, 2023

hpe-ykoehler commented on Mar 23, 2023

chrmarti commented on Mar 24, 2023

2 remaining items

chrmarti commented on Apr 3, 2023

TheButlah commented on May 17, 2023

chrmarti commented on May 19, 2023

hpe-ykoehler commented on May 19, 2023

TheButlah commented on May 19, 2023

hpe-ykoehler commented on Aug 24, 2023

andronat commented on Aug 29, 2023

randomthingsandstuff commented on Apr 22, 2024

trxcllnt commented on May 13, 2024

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions