Question: confusing behavior of built-in command completions like `cd`

[CicadaSeventeen]

Hello,

I and a few colleagues are currently working on adding Unicode (i.e., non-ASCII) natural language file/directory completion support for Bash and other shells.

Our method involves "hacking" (redefining and overriding) functions such as _comp_compgen_filedir or _comp_compgen__call_builtin. You can view a few existing solutions here:

• https://github.com/calsys456/bash-pinyin-completion-cpp/blob/main/scripts/bash_pinyin_completion
• https://github.com/muxueqz/bash-completion-pinyin.nim/blob/master/pinyin.completion
• https://github.com/AOSC-Dev/bash-pinyin-completion-rs/blob/main/scripts/bash_pinyin_completion

This approach currently works in environments using only bash-completion (without ble.sh), functioning correctly for both external commands and built-in commands being completed.

However, when using ble.sh, only external commands like ls and rmdir are completed correctly (while basic auto-completion functions as expected, see attached image). Built-in commands such as cd and pushd (which are highlighted as red by default in ble.sh), do not utilise our custom completion method. I also noted that other built-in commands (highlighted as purple by default in ble.sh) do not have this issue and work correctly with our custom completion.

above: only read named cd not works here

I am unsure what is special about the completion mechanism for built-in commands highlighted in red in ble.sh. Does it employ an implementation method different from standard bash-completion? I attempted to locate the relevant points by searching the source code but, unfortunately, was unsuccessful. I would greatly appreciate any explanation or assistance you can offer.

Thank you.

---

P.S. I know your native language is Japanese. I have heard that some people in Japan also desire similar completion functionality. I am personally re-implementing another Unicode completion scheme (see https://github.com/CicadaSeventeen/polLingua_Completion), which, unlike other implementations, focuses more on the possibility of supporting multiple natural languages instead of a single language. Although Japanese support is temporarily unavailable due to some difficulties, it should be supported in the future. However, this is another off-topic discussion.

[akinomyoga]

Does it employ an implementation method different from standard bash-completion?

Yes, they are implemented in the following part using the pure ble.sh API for completion (instead of the complete builtin of Bash).

ble.sh/lib/core-complete.sh

Lines 10531 to 10742 in 2f564e6

	# action:cdpath (action:file を修正)
	function ble/complete/action:cdpath/initialize {
	DATA=$cdpath_basedir
	ble/complete/action:file/initialize
	}
	function ble/complete/action:cdpath/complete {
	CAND=$DATA$CAND ble/complete/action:file/complete
	}
	function ble/complete/action:cdpath/init-menu-item {
	ble/color/face2g cmdinfo_cd_cdpath; g=$ret
	if [[ :$comp_type: == *:vstat:* ]]; then
	if [[ -h $CAND ]]; then
	suffix='@'
	elif [[ -d $CAND ]]; then
	suffix='/'
	fi
	fi
	}
	function ble/complete/action:cdpath/get-desc {
	local sgr0=$_ble_term_sgr0 sgr1= sgr2=
	local g ret g1 g2
	ble/syntax/highlight/getg-from-filename "$DATA$CAND"; g1=$g
	[[ $g1 ]] || { ble/color/face2g filename_warning; g1=$ret; }
	((g2=g1^_ble_color_gflags_Revert))
	ble/color/g2sgr "$g1"; sgr1=$ret
	ble/color/g2sgr "$g2"; sgr2=$ret
	ble/string#escape-for-display "$DATA$CAND" sgr1="$sgr2":sgr0="$sgr1"
	local filename=$sgr1$ret$sgr0
	CAND=$DATA$CAND ble/complete/action:file/get-desc
	desc="CDPATH $filename ($desc)"
	}
	function ble/cmdinfo/complete:cd/generate-cdable_vars {
	shopt -q cdable_vars || return 1
	ble/string#match "$COMPV" '^[_a-zA-Z0-9][_a-zA-Z0-9]*$' || return 1
	local arr
	ble/util/compgen arr -vX "_ble*" -- "$COMPV"
	ble/complete/source/test-limit "${#arr[@]}" || return 1
	local action=file old_cand_count=$cand_count
	local cand "${_ble_complete_yield_varnames[@]/%/=}" # WA #D1570 checked
	ble/complete/cand/yield.initialize "$action"
	for cand in "${arr[@]}"; do
	((cand_iloop++%bleopt_complete_polling_cycle==0)) && ble/complete/check-cancel && return 148
	[[ $cand == "$COMPV"?* && -d ${!cand-} ]] &&
	ble/complete/cand/yield "$action" "$cand" "$data"
	done
	((cand_count>old_cand_count))
	}
	## @fn ble/cmdinfo/complete:cd/.impl
	## @remarks
	## この実装は ble/complete/source:file を元にしている。
	## 実装に関する注意点はこの元の実装も参照の事。
	function ble/cmdinfo/complete:cd/.impl {
	local type=$1
	[[ $comps_flags == *v* ]] || return 1
	local old_cand_count=$cand_count
	case $type in
	(pushd|popd|dirs)
	# todo: -- より後の [-+]* は処理しない
	# todo: 実は -N/+N はオプションではなく通常引数
	if [[ $COMPV == [-+]* ]]; then
	# yield options
	local flags=n
	[[ $type == dirs ]] && flags=clpv
	ble/cmdinfo/complete/yield-flag "$type" "$flags" dedup:hasarg=0123456789:cancel-on-empty
	local "${_ble_complete_yield_varnames[@]/%/=}" # WA #D1570 checked
	ble/complete/cand/yield.initialize word
	local ret
	ble/color/face2sgr-ansi filename_directory
	local sgr1=$ret sgr0=$'\e[m'
	# yield -N/+N
	local i n=${#DIRSTACK[@]}
	for ((i=0;i<n;i++)); do
	local cand=${COMPV::1}$i
	[[ $cand == "$COMPV"* ]] || continue
	local j=$i; [[ $COMPV == -* ]] && j=$((n-1-i))
	ble/complete/cand/yield word "$cand" "DIRSTACK[$j] $sgr1${DIRSTACK[j]}$sgr0"
	done
	# yield - and -- for pushd
	if [[ $type == pushd ]]; then
	[[ ${OLDPWD:-} && $COMPV == - ]] &&
	ble/complete/cand/yield word - "OLDPWD $sgr1$OLDPWD$sgr0"
	[[ -- == "$COMPV"* ]] &&
	ble/complete/cand/yield word -- '(indicate the end of options)'
	fi
	((cand_count!=old_cand_count)) && return 0
	fi
	[[ $type == pushd ]] || return 0 ;;
	(*)
	# todo: -- より後の [-+]* は処理しない
	if [[ $COMPV == -* ]]; then
	local list=LP
	((_ble_bash>=40200)) && list=${list}e
	((_ble_bash>=40300)) && list=${list}@
	ble/cmdinfo/complete/yield-flag cd "$list" dedup
	local "${_ble_complete_yield_varnames[@]/%/=}" # WA #D1570 checked
	ble/complete/cand/yield.initialize word
	if [[ ${OLDPWD:-} && $COMPV == - ]]; then
	local ret
	ble/color/face2sgr-ansi filename_directory
	local sgr1=$ret sgr0=$'\e[m'
	ble/complete/cand/yield word - "OLDPWD $sgr1$OLDPWD$sgr0"
	fi
	[[ -- == "$COMPV"* ]] &&
	ble/complete/cand/yield word -- '(indicate the end of options)'
	return 0
	fi
	esac
	[[ :$comp_type: != *:[maA]:* && $COMPV =~ ^.+/ ]] && COMP_PREFIX=${BASH_REMATCH[0]}
	[[ :$comp_type: == *:[maA]:* && ! $COMPV ]] && return 1
	if [[ ! $CDPATH ]]; then
	ble/complete/source:dir || return "$?"
	((cand_count>old_cand_count)) && return 0
	ble/cmdinfo/complete:cd/generate-cdable_vars
	return "$?"
	fi
	ble/complete/source:tilde; local ext=$?
	((ext==148||ext==0)) && return "$ext"
	local is_pwd_visited= is_cdpath_generated=
	"${_ble_util_set_declare[@]//NAME/visited}" # WA #D1570 checked
	# Check CDPATH first
	local name names; ble/string#split names : "$CDPATH"
	for name in "${names[@]}"; do
	[[ $name ]] || continue
	name=${name%/}/
	# カレントディレクトリが CDPATH に含まれている時は action=file で登録
	local action=cdpath
	[[ ${name%/} == . || ${name%/} == "${PWD%/}" ]] &&
	is_pwd_visited=1 action=file
	local -a candidates=()
	local ret cand
	ble/complete/source:file/.construct-pathname-pattern "$COMPV"
	ble/complete/util/eval-pathname-expansion "$name$ret"; (($?==148)) && return 148
	ble/complete/source/test-limit "${#ret[@]}" || return 1
	for cand in "${ret[@]}"; do
	((cand_iloop++%bleopt_complete_polling_cycle==0)) &&
	ble/complete/check-cancel && return 148
	[[ $cand && -d $cand ]] || continue
	[[ $cand == / ]] || cand=${cand%/}
	cand=${cand#"$name"}
	ble/set#contains visited "$cand" && continue
	ble/set#add visited "$cand"
	ble/array#push candidates "$cand"
	done
	((${#candidates[@]})) || continue
	local flag_source_filter=1
	local cdpath_basedir=$name
	ble/complete/cand/yield-filenames "$action" "${candidates[@]}"; local ext=$?
	((ext==148)) && return "$ext"
	[[ $action == cdpath ]] && is_cdpath_generated=1
	done
	[[ $is_cdpath_generated ]] &&
	bleopt complete_menu_style=desc
	# Check PWD next
	# カレントディレクトリが CDPATH に含まれていなかった時に限り通常の候補生成
	if [[ ! $is_pwd_visited ]]; then
	local -a candidates=()
	local ret cand
	ble/complete/source:file/.construct-pathname-pattern "$COMPV"
	ble/complete/util/eval-pathname-expansion "${ret%/}/"; (($?==148)) && return 148
	ble/complete/source/test-limit "${#ret[@]}" || return 1
	for cand in "${ret[@]}"; do
	((cand_iloop++%bleopt_complete_polling_cycle==0)) &&
	ble/complete/check-cancel && return 148
	[[ -d $cand ]] || continue
	[[ $cand == / ]] || cand=${cand%/}
	ble/set#contains visited "$cand" && continue
	ble/array#push candidates "$cand"
	done
	local flag_source_filter=1
	ble/complete/cand/yield-filenames file "${candidates[@]}"; local ext=$?
	((ext==148)) && return "$ext"
	fi
	((cand_count>old_cand_count)) && return 0
	ble/cmdinfo/complete:cd/generate-cdable_vars
	}
	function ble/cmdinfo/complete:cd {
	ble/cmdinfo/complete:cd/.impl cd
	}
	function ble/cmdinfo/complete:pushd {
	ble/cmdinfo/complete:cd/.impl pushd
	}
	function ble/cmdinfo/complete:popd {
	ble/cmdinfo/complete:cd/.impl popd
	}
	function ble/cmdinfo/complete:dirs {
	ble/cmdinfo/complete:cd/.impl dirs
	}

P.S. I know your native language is Japanese. I have heard that some people in Japan also desire similar completion functionality.

Yeah, I also thought about a similar functionality and am still interested in it.

Although Japanese support is temporarily unavailable due to some difficulties, it should be supported in the future.

To get the romanized reading of a Japanese text, one may consider using a library of a category called 日本語形態素解析器, (though there seem to be many similar tools recently, and I'm not sure which one is better). A tricky part is that a single Japanese text may have multiple readings depending on the context, so one needs to find an analyzer that supports outputting multiple candidates.

[akinomyoga]

A tricky part is that a single Japanese text may have multiple readings depending on the context,

I installed the most standard one, MeCab (mecab-git and mecab-ipadic-git), though I needed to edit Makefile for mecab-ipadic-git as described in the comment. With the following command, one may generate five possible analysis results in katakana (片仮名). Although the reading on the second line is wrong, it shouldn't affect the use case for generating possible candidates in shell completion. Then, the mapping from katakana to romaji (the alphabetical representation, like 拼音 in Chinese) is simple.

$ mecab -N 5 -O yomi <<< '関西学院大学' | uniq
カンセイガクインダイガク
カンセイガクインダイマナブ

[CicadaSeventeen]

Thanks a lot. I will make a update later

[CicadaSeventeen]

Hello. I have been working on it and now pinyin completion of commands like cd or pushd works. However, if I type in bash, for example

./tp

The pinyin completion does not work.

If this case is somehow different?

[akinomyoga]

It's obvious that it's different. bash-completion doesn't provide completion for the command name, and the code that I pointed in the previous reply is clearly unrelated to the completion of command names. Please check the shell function ble/complete/source:command/.print-command in lib/core-compolete.sh.

[CicadaSeventeen]

Ops, I may have not explained clearly. ./tp here is not a command name. It is a file/dir name. That is like, when you type ./ and then type some string about files under current path ( PWD ), tab completion will provide completion for files.

I was talking about things like that.

[akinomyoga]

The answer is essentially the same. Even if it is the directory name, the completion of the directory name at that position is not provided by bash-completion, so patching bash-completion doesn't affect it. It is obvious that you should look for another place. Within the ble.sh codebase, the completion of the directory name at the command position is handled in the shell function ble/complete/source:command/.print in lib/core-complete.sh.