Linux 如何替换复杂文本文件中的 shell 变量

How to substitute shell variables in complex text files

提问by Ben

I have several text files in which I have introduced shell variables ($VAR1 or $VAR2 for instance).

我有几个文本文件，我在其中引入了 shell 变量（例如 $VAR1 或 $VAR2）。

I would like to take those files (one by one) and save them in new files where all variables would have been replaced.

我想将这些文件（一个接一个）保存在新文件中，所有变量都将被替换。

To do this, I used the following shell script (found on StackOverflow):

为此，我使用了以下 shell 脚本（在 StackOverflow 上找到）：

while read line
do
    eval echo "$line" >> destination.txt
done < "source.txt"

This works very well on very basic files.

这对非常基本的文件非常有效。

But on more complex files, the "eval" command does too much:

但是在更复杂的文件上，“eval”命令做的太多了：

• Lines starting with "#" are skipped

• XML files parsing results in tons of errors

• 跳过以“#”开头的行

• XML 文件解析导致大量错误

Is there a better way to do it? (in shell script... I know this is easily done with Ant for instance)

有没有更好的方法来做到这一点？（在 shell 脚本中......我知道这很容易用 Ant 完成）

Kind regards

亲切的问候

采纳答案by derobert

Looking, it turns out on my system there is an envsubstcommand which is part of the gettext-base package.

看，原来在我的系统上有一个envsubst命令，它是 gettext-base 包的一部分。

So, this makes it easy:

所以，这很容易：

envsubst < "source.txt" > "destination.txt"

Note if you want to use the same file for both, you'll have to use something like moreutil's sponge, as suggested by Johnny Utahh: envsubst < "source.txt" | sponge "source.txt". (Because the shell redirect will otherwise empty the file before its read.)

请注意，如果你想使用两个相同的文件，你必须使用类似moreutil的sponge，由约翰尼Utahh的建议：envsubst < "source.txt" | sponge "source.txt"。（因为 shell 重定向将在读取文件之前清空文件。）

回答by w00t

If you really only want to use bash (and sed), then I would go through each of your environment variables (as returned by setin posix mode) and build a bunch of -e 'regex'for sed from that, terminated by a -e 's/\$[a-zA-Z_][a-zA-Z0-9_]*//g', then pass all that to sed.

如果你真的只想使用 bash（和 sed），那么我会检查你的每个环境变量（set在 posix 模式下返回）并-e 'regex'从中构建一堆for sed，以 a 终止-e 's/\$[a-zA-Z_][a-zA-Z0-9_]*//g'，然后将所有这些传递给sed。

Perl would do a nicer job though, you have access to the environment vars as an array and you can do executable replacements so you only match any environment variable once.

但是，Perl 会做得更好，您可以将环境变量作为数组访问，并且可以执行可执行替换，因此您只能匹配任何环境变量一次。

回答by w00t

Actually you need to change your readto read -rwhich will make it ignore backslashes.

实际上，您需要将您的更改read为read -r这将使其忽略反斜杠。

Also, you should escape quotes and backslashes. So

此外，您应该转义引号和反斜杠。所以

while read -r line; do
  line="${line//\/\\}"
  line="${line//\"/\\"}"
  line="${line//\`/\\`}"
  eval echo "\"$line\""
done > destination.txt < source.txt

Still a terrible way to do expansion though.

尽管如此，仍然是一种可怕的扩展方式。

回答by Michael

In reference to answer 2, when discussing envsubst, you asked:

关于答案 2，在讨论 envsubst 时，您问：

How can I make it work with the variables that are declared in my .sh script?

如何使其与在我的 .sh 脚本中声明的变量一起使用？

The answer is you simply need to export your variables before calling envsubst.

答案是您只需要在调用envsubst.

You can also limit the variable strings you want to replace in the input using the envsubstSHELL_FORMATargument (avoiding the unintended replacement of a string in the input with a common shell variable value - e.g. $HOME).

您还可以使用envsubstSHELL_FORMAT参数限制要在输入中替换的变量字符串（避免使用通用 shell 变量值意外替换输入中的字符串 - 例如$HOME）。

For instance:

例如：

export VAR1='somevalue' VAR2='someothervalue'
MYVARS='$VAR1:$VAR2'

envsubst "$MYVARS" <source.txt >destination.txt

Will replace all instances of $VAR1and $VAR2(and only VAR1and VAR2) in source.txtwith 'somevalue'and 'someothervalue'respectively.

将分别用和替换$VAR1和$VAR2（并且仅VAR1和VAR2）的所有实例。source.txt'somevalue''someothervalue'

回答by Betlista

envsubstseems exactly like something I wanted to use, but -voption surprised me a bit.

envsubst看起来就像我想使用的东西，但-v选项让我有点惊讶。

While envsubst < template.txtwas working fine, the same with option -vwas not working:

虽然envsubst < template.txt工作正常，但同样的 with 选项-v不起作用：

$ cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.1 (Maipo)
$ envsubst -V
envsubst (GNU gettext-runtime) 0.18.2
Copyright (C) 2003-2007 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Written by Bruno Haible.

As I wrote, this was not working:

正如我所写，这不起作用：

$ envsubst -v < template.txt
envsubst: missing arguments
$ cat template.txt | envsubst -v
envsubst: missing arguments

I had to do this to make it work:

我必须这样做才能使其工作：

TEXT=`cat template.txt`; envsubst -v "$TEXT"

Maybe it helps someone.

也许它可以帮助某人。

回答by Govind Kailas

Export all the needed variables and then use a perl onliner

导出所有需要的变量，然后使用 perl onliner

TEXT=$(echo "$TEXT"|perl -wpne 's#\${?(\w+)}?# $ENV{$1} // $& #ge;')

TEXT=$(echo "$TEXT"|perl -wpne 's#\${?(\w+)}?# $ENV{$1} // $& #ge;')

This will replace all the ENV variables present in TEXT with actual values. Quotes are also preserved :)

这将用实际值替换 TEXT 中存在的所有 ENV 变量。报价也被保留:)

回答by Dado

1. Define your ENV variable

1. 定义您的 ENV 变量

$ export MY_ENV_VAR=congratulation

1. Create template file (in.txt) with following content

2. 使用以下内容创建模板文件（in.txt）

$MY_ENV_VAR

You can also use all other ENV variables defined by your system like (in linux) $TERM, $SHELL, $HOME...

您还可以使用系统定义的所有其他 ENV 变量，例如（在 linux 中）$TERM、$SHELL、$HOME...

1. Run this command to raplace all env-variables in your in.txtfile and to write the result to out.txt

3. 运行此命令以替换in.txt文件中的所有环境变量并将结果写入out.txt

$ envsubst "`printf '${%s} ' $(sh -c "env|cut -d'=' -f1")`" < in.txt > out.txt

1. Check the content of out.txt file

4. 检查out.txt文件的内容

$ cat out.txt

and you should see "congratulation".

你应该看到“祝贺”。

回答by inetphantom

I know this topic is old, but I have a simpler working solution without export the variables. Can be a oneliner, but I prefer to split using \on line end.

我知道这个话题很老，但我有一个更简单的工作解决方案，无需导出变量。可以是单线，但我更喜欢\在线路末端拆分。

var1='myVar1'\
var2=2\
var3=${var1}\
envsubst '$var1,$var3' < "source.txt" > "destination.txt"

#         ^^^^^^^^^^^    ^^^^^^^^^^     ^^^^^^^^^^^^^^^
# define which to replace   input            output

The variables need to be defined to the same line as envsubstis to get considered as environment variables.

变量需要定义到与envsubst被视为环境变量相同的行。

The '$var1,$var3'is optional to only replace the specified ones. Imagine an input file containing ${VARIABLE_USED_BY_JENKINS}which should not be replaced.

该'$var1,$var3'是可选的，只更换指定的。想象一个包含${VARIABLE_USED_BY_JENKINS}不应被替换的输入文件。

回答by Arik

If you want env variables to be replaced in your source files while keeping all of the non env variables as they are, you can use the following command:

如果您希望在源文件中替换 env 变量，同时保留所有非 env 变量，您可以使用以下命令：

envsubst "$(printf '${%s} ' $(env | sed 's/=.*//'))" < source.txt > destination.txt

The syntax for replacing only specific variables is explained here. The command above is using a sub-shell to list all defined variables and then passing it to the envsubst

此处解释了仅替换特定变量的语法。上面的命令使用子 shell 列出所有定义的变量，然后将其传递给envsubst

So if there's a defined env variable called $NAME, and your source.txtfile looks like this:

因此，如果有一个已定义的 env 变量称为$NAME，并且您的source.txt文件如下所示：

Hello $NAME
Your balance is 123 ($USD)

The destination.txtwill be:

该destination.txt会是：

Hello Arik
Your balance is 123 ($USD)

Notice that the $NAMEis replaced and the $USDis left untouched

请注意，$NAME被替换并且$USD保持不变