mirror of
https://github.com/kuhyx/testsAndMisc.git
synced 2026-11-10 20:27:02 +01:00
Fix two runtime defects the analyze_repo.sh split introduced
41cc045passed bash -n, shellcheck, a function-set diff, a line-set diff and a stubbed source run -- and still shipped a script that aborted after language detection. Both bugs needed a real execution to see. 1. set -e. The eight `((LANG_FILES[x] > 0)) && HAS_Y=true` lines were top-level statements, where a false `&&` merely left $? non-zero. As the tail of detect_languages they became its return value, and the last one (java, false for most repos) killed the script. Now `|| true`. 2. Self-referencing nameref. `local -n LANG_CODE_FILES="$1"` pointing at a global of the same name makes bash 5.3 warn "circular name reference" on every access -- into this script's own output. The locals are now _code_files. The rename then hid the array from shellcheck (SC2034), since it was only named as a bare word. Fixed by passing it through CODE_FILES_MAP so the reference is real, not by suppressing. Verified against133fccfend to end on a fixture repo: same file set, every text artifact byte-identical, exit 0 both sides, zero warnings. Only cscope.out.in differs, in binary padding -- same input list, same 90112 bytes. The lesson for the recipe doc: a stubbed run sources the libs and never calls them, so it cannot reach a bug in a seam that passes state. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011w2Pj4Ek3ZNMQVYcFHgXDY
This commit is contained in:
parent
a84556ac2a
commit
936e0ec70d
@ -0,0 +1,50 @@
|
||||
{
|
||||
"intent": "Fix two runtime defects the analyze_repo.sh split introduced in 41cc045, both of which every static check passed clean, and verify the fix by running the script end to end against the pre-split original.",
|
||||
"scope": [
|
||||
"linux_configuration/scripts/single_use/utils/analyze_repo.sh",
|
||||
"linux_configuration/scripts/single_use/utils/lib/analyze_repo_separate.sh",
|
||||
"linux_configuration/scripts/single_use/utils/lib/analyze_repo_report.sh",
|
||||
"Non-goal: any further restructuring; the seam from 41cc045 is unchanged"
|
||||
],
|
||||
"changes": [
|
||||
"Renamed the nameref locals from LANG_CODE_FILES to _code_files in both libs. A nameref whose local name equals the global it points at is a self-reference: bash 5.3 still moves the data, but warns 'circular name reference' on every single access, and those warnings land in this script's user-facing output.",
|
||||
"Replaced the entry's bare `declare -A LANG_CODE_FILES` plus two literal call arguments with a CODE_FILES_MAP variable holding the name. After the rename shellcheck could no longer see the array as used (SC2034), because it was only ever mentioned as a bare word; passing it through a variable makes the reference real rather than suppressing the warning.",
|
||||
"Added `|| true` to the eight `((LANG_FILES[x] > 0)) && HAS_Y=true` lines in detect_languages. These were top-level statements before the split, where a false `((...)) &&` merely left $? non-zero. As the tail of a function they become its return value, and under `set -e` the last one -- java, false for most repositories -- aborted the entire script immediately after language detection."
|
||||
],
|
||||
"verification": [
|
||||
{
|
||||
"command": "bash <stubbed entry> <fixture repo> <outdir>, against the same run from a detached worktree at 133fccf (pre-split)",
|
||||
"result": "pass",
|
||||
"evidence": "Pre-split exits 0 and completes. The split as committed in 41cc045 exited 1 immediately after 'Files found by language', producing none of STEPs 5-8. After the fix it exits 0 and reaches the final summary, with zero 'circular name reference' warnings (was one per map access)."
|
||||
},
|
||||
{
|
||||
"command": "diff of every generated artifact, pre-split run vs fixed run",
|
||||
"result": "pass",
|
||||
"evidence": "Identical file set, and every text artifact byte-identical: tokei_stats, scc_stats, scc_complexity, code_identifiers, comment_words, grep_keywords, grep_function_calls, grep_imports, the per_language/* files, tags, ctags_symbols, ctags_kinds and cscope.files. Only cscope.out.in differs, in binary hash-table padding; its input list cscope.files is identical and both indexes are the same 90112 bytes."
|
||||
},
|
||||
{
|
||||
"command": "shellcheck analyze_repo.sh lib/analyze_repo_*.sh",
|
||||
"result": "pass",
|
||||
"evidence": "Zero findings, no suppression added. The SC2034 introduced by the rename was fixed by making the reference real, not by silencing it."
|
||||
},
|
||||
{
|
||||
"command": "audit of every wrapped function's last statement for the same set -e hazard",
|
||||
"result": "pass",
|
||||
"evidence": "detect_languages was the only one ending in a conditional. analyze_symbols and analyze_cscope end in 'fi', analyze_clang_ast and separate_code_and_comments in echo, report_languages in a command already carrying '|| true'. Confirmed by the end-to-end run reaching the final summary."
|
||||
},
|
||||
{
|
||||
"command": "bash ~/utils/scripts/check_file_length.sh",
|
||||
"result": "pass",
|
||||
"evidence": "entry 234, report 228, separate 193, symbols 112, tools 124, scan 86. All six still under the cap after the added lines."
|
||||
}
|
||||
],
|
||||
"risks": [
|
||||
"Both defects were invisible to bash -n, shellcheck, the function-set diff, the line-set diff and the stubbed source run -- that last one sources the libs but never calls the functions, so it could not reach either bug. The general lesson is that a stubbed run proves the source lines resolve and nothing more; only a real execution exercises a seam that passes state.",
|
||||
"The fixture repository used for the comparison contains one Python and one C file, so the java/ruby/go/rust/shell branches of separate_code_and_comments were not executed. They are structurally identical to the two that were, and the line-set diff shows they differ from the original only by the mechanical rename.",
|
||||
"install_missing_tools was stubbed for both runs, so the package-installation path remains unexecuted under decision 6. It is unchanged by this commit."
|
||||
],
|
||||
"rollback": [
|
||||
"git revert this commit to return to 41cc045's state -- but note that state is the broken one: the script aborts after language detection. A rollback should go to 133fccf, reverting the split itself.",
|
||||
"After rollback, re-run the script against a small fixture repo and confirm it exits 0 and prints the STEP 8 summary."
|
||||
]
|
||||
}
|
||||
@ -165,10 +165,13 @@ print_subheader "Processing source files (separating code from comments)..."
|
||||
|
||||
COMMENTS_TEMP=$(mktemp)
|
||||
trap 'rm -f "$COMMENTS_TEMP" /tmp/code_*.tmp 2>/dev/null' EXIT
|
||||
declare -A LANG_CODE_FILES
|
||||
# Named once and passed by name to both passes, so the map itself never becomes
|
||||
# a global that the libs reach for implicitly.
|
||||
CODE_FILES_MAP="LANG_CODE_FILES"
|
||||
declare -A "$CODE_FILES_MAP"
|
||||
|
||||
separate_code_and_comments LANG_CODE_FILES "$COMMENTS_TEMP"
|
||||
report_languages LANG_CODE_FILES "$COMMENTS_TEMP"
|
||||
separate_code_and_comments "$CODE_FILES_MAP" "$COMMENTS_TEMP"
|
||||
report_languages "$CODE_FILES_MAP" "$COMMENTS_TEMP"
|
||||
|
||||
#==============================================================================
|
||||
# STEP 5: ctags Symbol Analysis
|
||||
|
||||
@ -41,7 +41,7 @@ KEYWORDS_RUBY="BEGIN|END|alias|and|begin|break|case|class|def|defined|do|else|el
|
||||
# Takes the map built by separate_code_and_comments explicitly, so no mutable
|
||||
# state crosses the file boundary between the two passes.
|
||||
report_languages() {
|
||||
local -n LANG_CODE_FILES="$1"
|
||||
local -n _code_files="$1"
|
||||
local COMMENTS_TEMP="$2"
|
||||
local lang code_file keywords output_file lang_file CODE_TEMP
|
||||
|
||||
@ -60,8 +60,8 @@ report_languages() {
|
||||
LANG_KEYWORDS[java]="$KEYWORDS_JAVA"
|
||||
|
||||
# Analyze each language separately
|
||||
for lang in "${!LANG_CODE_FILES[@]}"; do
|
||||
code_file="${LANG_CODE_FILES[$lang]}"
|
||||
for lang in "${!_code_files[@]}"; do
|
||||
code_file="${_code_files[$lang]}"
|
||||
keywords="${LANG_KEYWORDS[$lang]}"
|
||||
output_file="$RESULTS_DIR/per_language/keywords_${lang}.txt"
|
||||
|
||||
@ -79,8 +79,8 @@ report_languages() {
|
||||
#------------------------------------------------------------------------------
|
||||
print_subheader "Per-Language Function Calls"
|
||||
|
||||
for lang in "${!LANG_CODE_FILES[@]}"; do
|
||||
code_file="${LANG_CODE_FILES[$lang]}"
|
||||
for lang in "${!_code_files[@]}"; do
|
||||
code_file="${_code_files[$lang]}"
|
||||
output_file="$RESULTS_DIR/per_language/functions_${lang}.txt"
|
||||
|
||||
if [ -f "$code_file" ] && [ -s "$code_file" ]; then
|
||||
@ -100,84 +100,84 @@ report_languages() {
|
||||
print_subheader "Per-Language Imports/Includes"
|
||||
|
||||
# C/C++ includes
|
||||
if [ -n "${LANG_CODE_FILES[c_cpp]}" ] && [ -s "${LANG_CODE_FILES[c_cpp]}" ]; then
|
||||
if [ -n "${_code_files[c_cpp]}" ] && [ -s "${_code_files[c_cpp]}" ]; then
|
||||
echo -e "${YELLOW}=== C/C++ Includes ===${NC}"
|
||||
ugrep -o '#include\s*[<"][^>"]+[>"]' "${LANG_CODE_FILES[c_cpp]}" 2>/dev/null |
|
||||
ugrep -o '#include\s*[<"][^>"]+[>"]' "${_code_files[c_cpp]}" 2>/dev/null |
|
||||
fast_count 30 |
|
||||
tee "$RESULTS_DIR/per_language/imports_c_cpp.txt"
|
||||
fi
|
||||
|
||||
# Python imports
|
||||
if [ -n "${LANG_CODE_FILES[python]}" ] && [ -s "${LANG_CODE_FILES[python]}" ]; then
|
||||
if [ -n "${_code_files[python]}" ] && [ -s "${_code_files[python]}" ]; then
|
||||
echo ""
|
||||
echo -e "${YELLOW}=== Python Imports ===${NC}"
|
||||
ugrep -o '^\s*(from\s+\S+\s+import\s+\S+|import\s+\S+)' "${LANG_CODE_FILES[python]}" 2>/dev/null |
|
||||
ugrep -o '^\s*(from\s+\S+\s+import\s+\S+|import\s+\S+)' "${_code_files[python]}" 2>/dev/null |
|
||||
sed 's/^\s*//' |
|
||||
fast_count 30 |
|
||||
tee "$RESULTS_DIR/per_language/imports_python.txt"
|
||||
fi
|
||||
|
||||
# JavaScript imports
|
||||
if [ -n "${LANG_CODE_FILES[javascript]}" ] && [ -s "${LANG_CODE_FILES[javascript]}" ]; then
|
||||
if [ -n "${_code_files[javascript]}" ] && [ -s "${_code_files[javascript]}" ]; then
|
||||
echo ""
|
||||
echo -e "${YELLOW}=== JavaScript Imports ===${NC}"
|
||||
ugrep -o "(import\s+.*\s+from\s+['\"][^'\"]+['\"]|require\s*\(['\"][^'\"]+['\"]\))" "${LANG_CODE_FILES[javascript]}" 2>/dev/null |
|
||||
ugrep -o "(import\s+.*\s+from\s+['\"][^'\"]+['\"]|require\s*\(['\"][^'\"]+['\"]\))" "${_code_files[javascript]}" 2>/dev/null |
|
||||
fast_count 30 |
|
||||
tee "$RESULTS_DIR/per_language/imports_javascript.txt"
|
||||
fi
|
||||
|
||||
# TypeScript imports
|
||||
if [ -n "${LANG_CODE_FILES[typescript]}" ] && [ -s "${LANG_CODE_FILES[typescript]}" ]; then
|
||||
if [ -n "${_code_files[typescript]}" ] && [ -s "${_code_files[typescript]}" ]; then
|
||||
echo ""
|
||||
echo -e "${YELLOW}=== TypeScript Imports ===${NC}"
|
||||
ugrep -o "(import\s+.*\s+from\s+['\"][^'\"]+['\"]|require\s*\(['\"][^'\"]+['\"]\))" "${LANG_CODE_FILES[typescript]}" 2>/dev/null |
|
||||
ugrep -o "(import\s+.*\s+from\s+['\"][^'\"]+['\"]|require\s*\(['\"][^'\"]+['\"]\))" "${_code_files[typescript]}" 2>/dev/null |
|
||||
fast_count 30 |
|
||||
tee "$RESULTS_DIR/per_language/imports_typescript.txt"
|
||||
fi
|
||||
|
||||
# Go imports
|
||||
if [ -n "${LANG_CODE_FILES[go]}" ] && [ -s "${LANG_CODE_FILES[go]}" ]; then
|
||||
if [ -n "${_code_files[go]}" ] && [ -s "${_code_files[go]}" ]; then
|
||||
echo ""
|
||||
echo -e "${YELLOW}=== Go Imports ===${NC}"
|
||||
ugrep -o '"[^"]+/[^"]+"' "${LANG_CODE_FILES[go]}" 2>/dev/null |
|
||||
ugrep -o '"[^"]+/[^"]+"' "${_code_files[go]}" 2>/dev/null |
|
||||
fast_count 30 |
|
||||
tee "$RESULTS_DIR/per_language/imports_go.txt"
|
||||
fi
|
||||
|
||||
# Rust use statements
|
||||
if [ -n "${LANG_CODE_FILES[rust]}" ] && [ -s "${LANG_CODE_FILES[rust]}" ]; then
|
||||
if [ -n "${_code_files[rust]}" ] && [ -s "${_code_files[rust]}" ]; then
|
||||
echo ""
|
||||
echo -e "${YELLOW}=== Rust Use Statements ===${NC}"
|
||||
ugrep -o '^\s*use\s+[^;]+' "${LANG_CODE_FILES[rust]}" 2>/dev/null |
|
||||
ugrep -o '^\s*use\s+[^;]+' "${_code_files[rust]}" 2>/dev/null |
|
||||
sed 's/^\s*//' |
|
||||
fast_count 30 |
|
||||
tee "$RESULTS_DIR/per_language/imports_rust.txt"
|
||||
fi
|
||||
|
||||
# Java imports
|
||||
if [ -n "${LANG_CODE_FILES[java]}" ] && [ -s "${LANG_CODE_FILES[java]}" ]; then
|
||||
if [ -n "${_code_files[java]}" ] && [ -s "${_code_files[java]}" ]; then
|
||||
echo ""
|
||||
echo -e "${YELLOW}=== Java Imports ===${NC}"
|
||||
ugrep -o '^\s*import\s+[^;]+' "${LANG_CODE_FILES[java]}" 2>/dev/null |
|
||||
ugrep -o '^\s*import\s+[^;]+' "${_code_files[java]}" 2>/dev/null |
|
||||
sed 's/^\s*//' |
|
||||
fast_count 30 |
|
||||
tee "$RESULTS_DIR/per_language/imports_java.txt"
|
||||
fi
|
||||
|
||||
# Ruby requires
|
||||
if [ -n "${LANG_CODE_FILES[ruby]}" ] && [ -s "${LANG_CODE_FILES[ruby]}" ]; then
|
||||
if [ -n "${_code_files[ruby]}" ] && [ -s "${_code_files[ruby]}" ]; then
|
||||
echo ""
|
||||
echo -e "${YELLOW}=== Ruby Requires ===${NC}"
|
||||
ugrep -o "(require\s+['\"][^'\"]+['\"]|require_relative\s+['\"][^'\"]+['\"])" "${LANG_CODE_FILES[ruby]}" 2>/dev/null |
|
||||
ugrep -o "(require\s+['\"][^'\"]+['\"]|require_relative\s+['\"][^'\"]+['\"])" "${_code_files[ruby]}" 2>/dev/null |
|
||||
fast_count 30 |
|
||||
tee "$RESULTS_DIR/per_language/imports_ruby.txt"
|
||||
fi
|
||||
|
||||
# Shell sources
|
||||
if [ -n "${LANG_CODE_FILES[shell]}" ] && [ -s "${LANG_CODE_FILES[shell]}" ]; then
|
||||
if [ -n "${_code_files[shell]}" ] && [ -s "${_code_files[shell]}" ]; then
|
||||
echo ""
|
||||
echo -e "${YELLOW}=== Shell Sources ===${NC}"
|
||||
ugrep -o '(source\s+[^\s]+|\.\s+[^\s]+)' "${LANG_CODE_FILES[shell]}" 2>/dev/null |
|
||||
ugrep -o '(source\s+[^\s]+|\.\s+[^\s]+)' "${_code_files[shell]}" 2>/dev/null |
|
||||
fast_count 30 |
|
||||
tee "$RESULTS_DIR/per_language/imports_shell.txt"
|
||||
fi
|
||||
@ -189,7 +189,7 @@ report_languages() {
|
||||
|
||||
# Create combined CODE_TEMP
|
||||
CODE_TEMP=$(mktemp)
|
||||
for lang_file in "${LANG_CODE_FILES[@]}"; do
|
||||
for lang_file in "${_code_files[@]}"; do
|
||||
[ -f "$lang_file" ] && cat "$lang_file" >>"$CODE_TEMP"
|
||||
done
|
||||
|
||||
|
||||
@ -12,9 +12,6 @@
|
||||
# separate_code_and_comments below. Keeping definition and use in one file is
|
||||
# what lets shellcheck see them as used.
|
||||
|
||||
# separate_code_and_comments <code_files_map_name> <comments_file>
|
||||
# The map is filled by nameref rather than left in a global, so no mutable
|
||||
# state crosses the file boundary into the reporting pass.
|
||||
# Populates LANG_FILES and the HAS_* family from the files present in the cwd.
|
||||
detect_languages() {
|
||||
print_subheader "Detecting languages in repository..."
|
||||
@ -62,18 +59,28 @@ detect_languages() {
|
||||
HAS_RUST=false
|
||||
HAS_JAVA=false
|
||||
|
||||
((LANG_FILES[c] + LANG_FILES[cpp] + LANG_FILES[h] > 0)) && HAS_C_FAMILY=true
|
||||
((LANG_FILES[python] > 0)) && HAS_PYTHON=true
|
||||
((LANG_FILES[javascript] + LANG_FILES[typescript] > 0)) && HAS_JS_FAMILY=true
|
||||
((LANG_FILES[shell] > 0)) && HAS_SHELL=true
|
||||
((LANG_FILES[ruby] > 0)) && HAS_RUBY=true
|
||||
((LANG_FILES[go] > 0)) && HAS_GO=true
|
||||
((LANG_FILES[rust] > 0)) && HAS_RUST=true
|
||||
((LANG_FILES[java] > 0)) && HAS_JAVA=true
|
||||
# `|| true` on each: these were top-level statements before the split, where
|
||||
# a false `((...)) &&` merely left $? non-zero. As the tail of a function
|
||||
# they become its return value, and under `set -e` the last one (java, false
|
||||
# for most repos) aborted the whole script.
|
||||
((LANG_FILES[c] + LANG_FILES[cpp] + LANG_FILES[h] > 0)) && HAS_C_FAMILY=true || true
|
||||
((LANG_FILES[python] > 0)) && HAS_PYTHON=true || true
|
||||
((LANG_FILES[javascript] + LANG_FILES[typescript] > 0)) && HAS_JS_FAMILY=true || true
|
||||
((LANG_FILES[shell] > 0)) && HAS_SHELL=true || true
|
||||
((LANG_FILES[ruby] > 0)) && HAS_RUBY=true || true
|
||||
((LANG_FILES[go] > 0)) && HAS_GO=true || true
|
||||
((LANG_FILES[rust] > 0)) && HAS_RUST=true || true
|
||||
((LANG_FILES[java] > 0)) && HAS_JAVA=true || true
|
||||
}
|
||||
|
||||
# separate_code_and_comments <code_files_map_name> <comments_file>
|
||||
# The map is filled by nameref rather than left in a global, so no mutable
|
||||
# state crosses the file boundary into the reporting pass. The local is named
|
||||
# _code_files, deliberately NOT the caller's name: a nameref whose local name
|
||||
# equals the variable it points at makes bash warn "circular name reference" on
|
||||
# every access, which would land in this script's user-facing output.
|
||||
separate_code_and_comments() {
|
||||
local -n LANG_CODE_FILES="$1"
|
||||
local -n _code_files="$1"
|
||||
local COMMENTS_TEMP="$2"
|
||||
|
||||
# Create per-language output directory
|
||||
@ -82,27 +89,27 @@ separate_code_and_comments() {
|
||||
# Process C/C++ files
|
||||
if $HAS_C_FAMILY; then
|
||||
echo "Processing C/C++ files..."
|
||||
LANG_CODE_FILES["c_cpp"]=$(mktemp /tmp/code_c_cpp.XXXXXX.tmp)
|
||||
find_files "*.c" "*.cpp" "*.cc" "*.cxx" "*.h" "*.hpp" | head -15000 | xargs cat 2>/dev/null >"${LANG_CODE_FILES[c_cpp]}"
|
||||
_code_files["c_cpp"]=$(mktemp /tmp/code_c_cpp.XXXXXX.tmp)
|
||||
find_files "*.c" "*.cpp" "*.cc" "*.cxx" "*.h" "*.hpp" | head -15000 | xargs cat 2>/dev/null >"${_code_files[c_cpp]}"
|
||||
|
||||
# Extract and strip C-style comments
|
||||
perl -0777 -ne 'while (/\/\*(.+?)\*\//gs) { print "$1\n"; } while (/\/\/([^\n]*)/g) { print "$1\n"; }' "${LANG_CODE_FILES[c_cpp]}" >>"$COMMENTS_TEMP"
|
||||
perl -0777 -pe 's|/\*.*?\*/||gs; s|//[^\n]*||g;' "${LANG_CODE_FILES[c_cpp]}" >"${LANG_CODE_FILES[c_cpp]}.clean"
|
||||
mv "${LANG_CODE_FILES[c_cpp]}.clean" "${LANG_CODE_FILES[c_cpp]}"
|
||||
perl -0777 -ne 'while (/\/\*(.+?)\*\//gs) { print "$1\n"; } while (/\/\/([^\n]*)/g) { print "$1\n"; }' "${_code_files[c_cpp]}" >>"$COMMENTS_TEMP"
|
||||
perl -0777 -pe 's|/\*.*?\*/||gs; s|//[^\n]*||g;' "${_code_files[c_cpp]}" >"${_code_files[c_cpp]}.clean"
|
||||
mv "${_code_files[c_cpp]}.clean" "${_code_files[c_cpp]}"
|
||||
fi
|
||||
|
||||
# Process JavaScript files (separate from TypeScript)
|
||||
if $HAS_JS_FAMILY; then
|
||||
echo "Processing JavaScript files..."
|
||||
LANG_CODE_FILES["javascript"]=$(mktemp /tmp/code_js.XXXXXX.tmp)
|
||||
find_files "*.js" "*.jsx" | head -15000 | xargs cat 2>/dev/null >"${LANG_CODE_FILES[javascript]}"
|
||||
_code_files["javascript"]=$(mktemp /tmp/code_js.XXXXXX.tmp)
|
||||
find_files "*.js" "*.jsx" | head -15000 | xargs cat 2>/dev/null >"${_code_files[javascript]}"
|
||||
|
||||
echo "Processing TypeScript files..."
|
||||
LANG_CODE_FILES["typescript"]=$(mktemp /tmp/code_ts.XXXXXX.tmp)
|
||||
find_files "*.ts" "*.tsx" | head -15000 | xargs cat 2>/dev/null >"${LANG_CODE_FILES[typescript]}"
|
||||
_code_files["typescript"]=$(mktemp /tmp/code_ts.XXXXXX.tmp)
|
||||
find_files "*.ts" "*.tsx" | head -15000 | xargs cat 2>/dev/null >"${_code_files[typescript]}"
|
||||
|
||||
# Extract and strip comments from both
|
||||
for lang_file in "${LANG_CODE_FILES[javascript]}" "${LANG_CODE_FILES[typescript]}"; do
|
||||
for lang_file in "${_code_files[javascript]}" "${_code_files[typescript]}"; do
|
||||
[ ! -s "$lang_file" ] && continue
|
||||
perl -0777 -ne 'while (/\/\*(.+?)\*\//gs) { print "$1\n"; } while (/\/\/([^\n]*)/g) { print "$1\n"; }' "$lang_file" >>"$COMMENTS_TEMP"
|
||||
perl -0777 -pe 's|/\*.*?\*/||gs; s|//[^\n]*||g;' "$lang_file" >"${lang_file}.clean"
|
||||
@ -113,74 +120,74 @@ separate_code_and_comments() {
|
||||
# Process Python files
|
||||
if $HAS_PYTHON; then
|
||||
echo "Processing Python files..."
|
||||
LANG_CODE_FILES["python"]=$(mktemp /tmp/code_python.XXXXXX.tmp)
|
||||
find_files "*.py" | head -15000 | xargs cat 2>/dev/null >"${LANG_CODE_FILES[python]}"
|
||||
_code_files["python"]=$(mktemp /tmp/code_python.XXXXXX.tmp)
|
||||
find_files "*.py" | head -15000 | xargs cat 2>/dev/null >"${_code_files[python]}"
|
||||
|
||||
perl -ne 'if (/^\s*#(.*)/) { print "$1\n"; } elsif (/#(.*)$/) { print "$1\n"; }' "${LANG_CODE_FILES[python]}" >>"$COMMENTS_TEMP"
|
||||
perl -0777 -ne 'while (/"""(.+?)"""/gs) { print "$1\n"; } while (/'"'"''"'"''"'"'(.+?)'"'"''"'"''"'"'/gs) { print "$1\n"; }' "${LANG_CODE_FILES[python]}" >>"$COMMENTS_TEMP"
|
||||
perl -pe 's/#.*$//' "${LANG_CODE_FILES[python]}" | perl -0777 -pe 's/""".*?"""//gs; s/'"'"''"'"''"'"'.*?'"'"''"'"''"'"'//gs' >"${LANG_CODE_FILES[python]}.clean"
|
||||
mv "${LANG_CODE_FILES[python]}.clean" "${LANG_CODE_FILES[python]}"
|
||||
perl -ne 'if (/^\s*#(.*)/) { print "$1\n"; } elsif (/#(.*)$/) { print "$1\n"; }' "${_code_files[python]}" >>"$COMMENTS_TEMP"
|
||||
perl -0777 -ne 'while (/"""(.+?)"""/gs) { print "$1\n"; } while (/'"'"''"'"''"'"'(.+?)'"'"''"'"''"'"'/gs) { print "$1\n"; }' "${_code_files[python]}" >>"$COMMENTS_TEMP"
|
||||
perl -pe 's/#.*$//' "${_code_files[python]}" | perl -0777 -pe 's/""".*?"""//gs; s/'"'"''"'"''"'"'.*?'"'"''"'"''"'"'//gs' >"${_code_files[python]}.clean"
|
||||
mv "${_code_files[python]}.clean" "${_code_files[python]}"
|
||||
fi
|
||||
|
||||
# Process Go files
|
||||
if $HAS_GO; then
|
||||
echo "Processing Go files..."
|
||||
LANG_CODE_FILES["go"]=$(mktemp /tmp/code_go.XXXXXX.tmp)
|
||||
find_files "*.go" | head -15000 | xargs cat 2>/dev/null >"${LANG_CODE_FILES[go]}"
|
||||
_code_files["go"]=$(mktemp /tmp/code_go.XXXXXX.tmp)
|
||||
find_files "*.go" | head -15000 | xargs cat 2>/dev/null >"${_code_files[go]}"
|
||||
|
||||
perl -0777 -ne 'while (/\/\*(.+?)\*\//gs) { print "$1\n"; } while (/\/\/([^\n]*)/g) { print "$1\n"; }' "${LANG_CODE_FILES[go]}" >>"$COMMENTS_TEMP"
|
||||
perl -0777 -pe 's|/\*.*?\*/||gs; s|//[^\n]*||g;' "${LANG_CODE_FILES[go]}" >"${LANG_CODE_FILES[go]}.clean"
|
||||
mv "${LANG_CODE_FILES[go]}.clean" "${LANG_CODE_FILES[go]}"
|
||||
perl -0777 -ne 'while (/\/\*(.+?)\*\//gs) { print "$1\n"; } while (/\/\/([^\n]*)/g) { print "$1\n"; }' "${_code_files[go]}" >>"$COMMENTS_TEMP"
|
||||
perl -0777 -pe 's|/\*.*?\*/||gs; s|//[^\n]*||g;' "${_code_files[go]}" >"${_code_files[go]}.clean"
|
||||
mv "${_code_files[go]}.clean" "${_code_files[go]}"
|
||||
fi
|
||||
|
||||
# Process Rust files
|
||||
if $HAS_RUST; then
|
||||
echo "Processing Rust files..."
|
||||
LANG_CODE_FILES["rust"]=$(mktemp /tmp/code_rust.XXXXXX.tmp)
|
||||
find_files "*.rs" | head -15000 | xargs cat 2>/dev/null >"${LANG_CODE_FILES[rust]}"
|
||||
_code_files["rust"]=$(mktemp /tmp/code_rust.XXXXXX.tmp)
|
||||
find_files "*.rs" | head -15000 | xargs cat 2>/dev/null >"${_code_files[rust]}"
|
||||
|
||||
perl -0777 -ne 'while (/\/\*(.+?)\*\//gs) { print "$1\n"; } while (/\/\/([^\n]*)/g) { print "$1\n"; }' "${LANG_CODE_FILES[rust]}" >>"$COMMENTS_TEMP"
|
||||
perl -0777 -pe 's|/\*.*?\*/||gs; s|//[^\n]*||g;' "${LANG_CODE_FILES[rust]}" >"${LANG_CODE_FILES[rust]}.clean"
|
||||
mv "${LANG_CODE_FILES[rust]}.clean" "${LANG_CODE_FILES[rust]}"
|
||||
perl -0777 -ne 'while (/\/\*(.+?)\*\//gs) { print "$1\n"; } while (/\/\/([^\n]*)/g) { print "$1\n"; }' "${_code_files[rust]}" >>"$COMMENTS_TEMP"
|
||||
perl -0777 -pe 's|/\*.*?\*/||gs; s|//[^\n]*||g;' "${_code_files[rust]}" >"${_code_files[rust]}.clean"
|
||||
mv "${_code_files[rust]}.clean" "${_code_files[rust]}"
|
||||
fi
|
||||
|
||||
# Process Ruby files
|
||||
if $HAS_RUBY; then
|
||||
echo "Processing Ruby files..."
|
||||
LANG_CODE_FILES["ruby"]=$(mktemp /tmp/code_ruby.XXXXXX.tmp)
|
||||
find_files "*.rb" | head -5000 | xargs cat 2>/dev/null >"${LANG_CODE_FILES[ruby]}"
|
||||
_code_files["ruby"]=$(mktemp /tmp/code_ruby.XXXXXX.tmp)
|
||||
find_files "*.rb" | head -5000 | xargs cat 2>/dev/null >"${_code_files[ruby]}"
|
||||
|
||||
perl -ne 'if (/#(.*)$/) { print "$1\n"; }' "${LANG_CODE_FILES[ruby]}" >>"$COMMENTS_TEMP"
|
||||
perl -0777 -ne 'while (/=begin(.+?)=end/gs) { print "$1\n"; }' "${LANG_CODE_FILES[ruby]}" >>"$COMMENTS_TEMP"
|
||||
perl -pe 's/#.*$//' "${LANG_CODE_FILES[ruby]}" | perl -0777 -pe 's/=begin.*?=end//gs' >"${LANG_CODE_FILES[ruby]}.clean"
|
||||
mv "${LANG_CODE_FILES[ruby]}.clean" "${LANG_CODE_FILES[ruby]}"
|
||||
perl -ne 'if (/#(.*)$/) { print "$1\n"; }' "${_code_files[ruby]}" >>"$COMMENTS_TEMP"
|
||||
perl -0777 -ne 'while (/=begin(.+?)=end/gs) { print "$1\n"; }' "${_code_files[ruby]}" >>"$COMMENTS_TEMP"
|
||||
perl -pe 's/#.*$//' "${_code_files[ruby]}" | perl -0777 -pe 's/=begin.*?=end//gs' >"${_code_files[ruby]}.clean"
|
||||
mv "${_code_files[ruby]}.clean" "${_code_files[ruby]}"
|
||||
fi
|
||||
|
||||
# Process Shell files
|
||||
if $HAS_SHELL; then
|
||||
echo "Processing Shell files..."
|
||||
LANG_CODE_FILES["shell"]=$(mktemp /tmp/code_shell.XXXXXX.tmp)
|
||||
find_files "*.sh" "*.bash" | head -5000 | xargs cat 2>/dev/null >"${LANG_CODE_FILES[shell]}"
|
||||
_code_files["shell"]=$(mktemp /tmp/code_shell.XXXXXX.tmp)
|
||||
find_files "*.sh" "*.bash" | head -5000 | xargs cat 2>/dev/null >"${_code_files[shell]}"
|
||||
|
||||
perl -ne 'if (/^\s*#(.*)/ && !/^#!/) { print "$1\n"; } elsif (/#(.*)$/) { print "$1\n"; }' "${LANG_CODE_FILES[shell]}" >>"$COMMENTS_TEMP"
|
||||
perl -pe 's/#.*$//' "${LANG_CODE_FILES[shell]}" >"${LANG_CODE_FILES[shell]}.clean"
|
||||
mv "${LANG_CODE_FILES[shell]}.clean" "${LANG_CODE_FILES[shell]}"
|
||||
perl -ne 'if (/^\s*#(.*)/ && !/^#!/) { print "$1\n"; } elsif (/#(.*)$/) { print "$1\n"; }' "${_code_files[shell]}" >>"$COMMENTS_TEMP"
|
||||
perl -pe 's/#.*$//' "${_code_files[shell]}" >"${_code_files[shell]}.clean"
|
||||
mv "${_code_files[shell]}.clean" "${_code_files[shell]}"
|
||||
fi
|
||||
|
||||
# Process Java files
|
||||
if $HAS_JAVA; then
|
||||
echo "Processing Java files..."
|
||||
LANG_CODE_FILES["java"]=$(mktemp /tmp/code_java.XXXXXX.tmp)
|
||||
find_files "*.java" | head -15000 | xargs cat 2>/dev/null >"${LANG_CODE_FILES[java]}"
|
||||
_code_files["java"]=$(mktemp /tmp/code_java.XXXXXX.tmp)
|
||||
find_files "*.java" | head -15000 | xargs cat 2>/dev/null >"${_code_files[java]}"
|
||||
|
||||
perl -0777 -ne 'while (/\/\*(.+?)\*\//gs) { print "$1\n"; } while (/\/\/([^\n]*)/g) { print "$1\n"; }' "${LANG_CODE_FILES[java]}" >>"$COMMENTS_TEMP"
|
||||
perl -0777 -pe 's|/\*.*?\*/||gs; s|//[^\n]*||g;' "${LANG_CODE_FILES[java]}" >"${LANG_CODE_FILES[java]}.clean"
|
||||
mv "${LANG_CODE_FILES[java]}.clean" "${LANG_CODE_FILES[java]}"
|
||||
perl -0777 -ne 'while (/\/\*(.+?)\*\//gs) { print "$1\n"; } while (/\/\/([^\n]*)/g) { print "$1\n"; }' "${_code_files[java]}" >>"$COMMENTS_TEMP"
|
||||
perl -0777 -pe 's|/\*.*?\*/||gs; s|//[^\n]*||g;' "${_code_files[java]}" >"${_code_files[java]}.clean"
|
||||
mv "${_code_files[java]}.clean" "${_code_files[java]}"
|
||||
fi
|
||||
|
||||
local COMMENT_LINES
|
||||
COMMENT_LINES=$(wc -l <"$COMMENTS_TEMP")
|
||||
echo ""
|
||||
echo "Processed languages: ${!LANG_CODE_FILES[*]}"
|
||||
echo "Processed languages: ${!_code_files[*]}"
|
||||
echo "Total comment lines: $COMMENT_LINES"
|
||||
}
|
||||
|
||||
Loading…
Reference in New Issue
Block a user