Remove duplicate $PATH entries with awk command

I am attempting to write a bash shell function that will certainly permit me to remove replicate copies of directory sites from my PATH setting variable.

I was informed that it is feasible to attain this with a one line command making use of the awk command, yet I can not identify just how to do it. Any person recognize just how?

2022-06-07 14:37:22
Source Share
Answers: 9

I would certainly do it simply with standard devices such as tr, type and also uniq:

NEW_PATH=`echo $PATH | tr ':' '\n' | sort | uniq | tr '\n' ':'`

If there is second best or unusual in your path it need to function

2022-07-14 00:19:32

This is my version:

path_no_dup () 
    local IFS=: p=();

    while read -r; do
    done < <(sort -u <(read -ra arr <<< "$1" && printf '%s\n' "${arr[@]}"));

    # Do whatever you like with "${p[*]}"
    echo "${p[*]}"

Usage: path_no_dup "$PATH"

Sample output:

rany$ v='a:a:a:b:b:b:c:c:c:a:a:a:b:c:a'; path_no_dup "$v"
2022-07-14 00:15:42
PATH=`awk -F: '{for (i=1;i<=NF;i++) { if ( !x[$i]++ ) printf("%s:",$i); }}' <<< "$PATH"`

Explanation of awk code:

  1. Separate the input by colons
  2. Append new path access to associative array for rapid replicate appearance - up
  3. Prints the associative array

In enhancement to being laconic, this set - lining is quickly: awk makes use of a chaining hash - table to attain amortized O (1) performance.

based upon Removing duplicate $PATH entries

2022-07-09 22:21:06
PATH=`perl -e 'print join ":", grep {!$h{$_}++} split ":", $ENV{PATH}'`
export PATH

This makes use of perl and also has numerous advantages:

  1. It gets rid of replicates
  2. It maintains type order
  3. It maintains the earliest look (/usr/bin:/sbin:/usr/bin will certainly cause /usr/bin:/sbin)
2022-06-08 06:24:39

Use awk to divide the path on :, after that loop over each area and also store it in an array. If you find an area which is currently in the array, that suggests you have actually seen it in the past, so do not publish it.

Below is an instance:

$ MYPATH=.:/foo/bar/bin:/usr/bin:/foo/bar/bin
$ awk -F: '{for(i=1;i<=NF;i++) if(!($i in arr)){arr[$i];printf s$i;s=":"}}' <<< "$MYPATH"

(Updated to remove the routing :.)

2022-06-07 15:09:09

There has actually been a comparable conversation concerning this here.

I take a little a various strategy. As opposed to simply approving the PATH that is set from all the various initialization documents that get mounted, I favor making use of getconf to recognize the system path and also area it first, after that add my recommended path order, after that make use of awk to remove any kind of matches. This might or might not actually quicken command implementation (and also theoretically be extra safe and secure), yet it offers me cozy fuzzies.

# I am entering my preferred PATH order here because it gets set,
# appended, reset, appended again and ends up in such a jumbled order.
# The duplicates get removed, preserving my preferred order.
PATH=$(command -p getconf PATH):/sbin:/usr/sbin:/usr/local/bin:/usr/local/sbin:$PATH
# Remove duplicates
PATH="$(printf "%s" "${PATH}" | /usr/bin/awk -v RS=: -v ORS=: '!($0 in a) {a[$0]; print}')"
export PATH

[~]$ echo $PATH
2022-06-07 15:07:25

Also sed (below making use of GNU sed syntax) can do the work:

MYPATH=$(printf '%s\n' "$MYPATH" | sed ':b;s/:\([^:]*\)\(:.*\):\1/:\1\2/;tb')

this set functions well just in instance first path is . like in dogbane is instance.

As a whole instance you require to add yet an additional s command:

MYPATH=$(printf '%s\n' "$MYPATH" | sed ':b;s/:\([^:]*\)\(:.*\):\1/:\1\2/;tb;s/^\([^:]*\)\(:.*\):\1/:\1\2/')

It functions also on such building and construction:

$ echo "/bin:.:/foo/bar/bin:/usr/bin:/foo/bar/bin:/foo/bar/bin:/bar/bin:/usr/bin:/bin" \
| sed ':b;s/:\([^:]*\)\(:.*\):\1/:\1\2/;tb;s/^\([^:]*\)\(:.*\):\1/\1\2/'

2022-06-07 15:04:05

Here is a streamlined one:

printf %s "$PATH" | awk -v RS=: -v ORS=: '!arr[$0]++'

Longer (to see just how it functions):

printf %s "$PATH" | awk -v RS=: -v ORS=: '{ if (!arr[$0]++) { print $0 } }'

Ok, given that you are new to linux, below is just how to in fact set PATH without a routing ": "

PATH=`printf %s "$PATH" | awk -v RS=: '{ if (!arr[$0]++) {printf("%s%s",!ln++?"":":",$0)}}'`

btw see to it to NOT have directory sites having ": " in your PATH, or else it is gon na be screwed up.

some debt to:

2022-06-07 15:02:12

If you do not currently have matches in the PATH and also you just intend to add directory sites if they are not currently there, you can do it conveniently with the shell alone.

for x in /path/to/add …; do
  case ":$PATH:" in
    *":$x:"*) :;; # already there
    *) PATH="$x:$PATH";;

And below is a shell fragment that gets rid of matches from $PATH. It experiences the access individually, and also duplicates those that have not been seen yet.

if [ -n "$PATH" ]; then
  old_PATH=$PATH:; PATH=
  while [ -n "$old_PATH" ]; do
    x=${old_PATH%%:*}       # the first remaining entry
    case $PATH: in
      *:"$x":*) ;;          # already there
      *) PATH=$PATH:$x;;    # not there yet
  unset old_PATH x
2022-06-07 14:43:20