Linux 从 HTML 表格中提取数据
声明：本页面是StackOverFlow热门问题的中英对照翻译，遵循CC BY-SA 4.0协议，如果您需要使用它，必须同样遵循CC BY-SA许可，注明原文地址和作者信息，同时你必须将它归于原作者（不是我）：StackOverFlow
原文地址: http://stackoverflow.com/questions/11790535/
Warning: these are provided under cc-by-sa 4.0 license. You are free to use/share it, But you must attribute it to the original authors (not me):
StackOverFlow
Extracting data from HTML table
提问by Gandalf StormCrow
I am looking for a way to get certain info from HTML in linux shell environment.
我正在寻找一种在 linux shell 环境中从 HTML 获取某些信息的方法。
This is bit that I'm interested in :
这是我感兴趣的一点：
<table class="details" border="0" cellpadding="5" cellspacing="2" width="95%">
<tr valign="top">
<th>Tests</th>
<th>Failures</th>
<th>Success Rate</th>
<th>Average Time</th>
<th>Min Time</th>
<th>Max Time</th>
</tr>
<tr valign="top" class="Failure">
<td>103</td>
<td>24</td>
<td>76.70%</td>
<td>71 ms</td>
<td>0 ms</td>
<td>829 ms</td>
</tr>
</table>
And I want to store in shell variables or echo these in key value pairs extracted from above html. Example :
我想存储在 shell 变量中或在从上面的 html 中提取的键值对中回显这些变量。例子 ：
Tests : 103
Failures : 24
Success Rate : 76.70 %
and so on..
What I can do at the moment is to create a java program that will use sax parser or html parser such as jsoup to extract this info.
我目前可以做的是创建一个 Java 程序，该程序将使用 sax 解析器或 html 解析器（例如 jsoup）来提取此信息。
But using java here seems to be overhead with including the runnable jar inside the "wrapper" script you want to execute.
但是在这里使用 java 似乎是在要执行的“包装器”脚本中包含可运行的 jar 的开销。
I'm sure that there must be "shell" languages out there that can do the same i.e. perl, python, bash etc.
我确信一定有“shell”语言可以做同样的事情，即 perl、python、bash 等。
My problem is that I have zero experience with these, can somebody help me resolve this "fairly easy" issue
我的问题是我对这些的经验为零，有人可以帮我解决这个“相当容易”的问题吗
Quick update:
快速更新：
I forgot to mention that I've got more tables and more rows in the .html document sorry about that (early morning).
我忘了提到我在 .html 文档中有更多的表格和更多的行，对此感到抱歉（清晨）。
Update #2:
更新#2：
Tried to install Bsoup like this since I don't have root access :
由于我没有 root 访问权限，因此尝试像这样安装 Bsoup：
$ wget http://www.crummy.com/software/BeautifulSoup/bs4/download/4.0/beautifulsoup4-4.1.0.tar.gz
$ tar -zxvf beautifulsoup4-4.1.0.tar.gz
$ cp -r beautifulsoup4-4.1.0/bs4 .
$ vi htmlParse.py # (paste code from ) Tichodromas' answer, just in case this (http://pastebin.com/4Je11Y9q) is what I pasted
$ run file (python htmlParse.py)
error:
错误：
$ python htmlParse.py
Traceback (most recent call last):
File "htmlParse.py", line 1, in ?
from bs4 import BeautifulSoup
File "/home/gdd/setup/py/bs4/__init__.py", line 29
from .builder import builder_registry
^
SyntaxError: invalid syntax
Update #3 :
更新 #3：
Running Tichodromas' answer get this error :
运行 Tichodromas 的回答得到这个错误：
Traceback (most recent call last):
File "test.py", line 27, in ?
headings = [th.get_text() for th in table.find("tr").find_all("th")]
TypeError: 'NoneType' object is not callable
any ideas?
有任何想法吗？
采纳答案by cdtits
A Python solution using BeautifulSoup4(Edit:with proper skipping. Edit3:Using class="details"to select the table):
使用A的Python溶液BeautifulSoup4（编辑：用适当的跳跃。EDIT3：使用class="details"选择table）：
from bs4 import BeautifulSoup
html = """
<table class="details" border="0" cellpadding="5" cellspacing="2" width="95%">
<tr valign="top">
<th>Tests</th>
<th>Failures</th>
<th>Success Rate</th>
<th>Average Time</th>
<th>Min Time</th>
<th>Max Time</th>
</tr>
<tr valign="top" class="Failure">
<td>103</td>
<td>24</td>
<td>76.70%</td>
<td>71 ms</td>
<td>0 ms</td>
<td>829 ms</td>
</tr>
</table>"""
soup = BeautifulSoup(html)
table = soup.find("table", attrs={"class":"details"})
# The first tr contains the field names.
headings = [th.get_text() for th in table.find("tr").find_all("th")]
datasets = []
for row in table.find_all("tr")[1:]:
dataset = zip(headings, (td.get_text() for td in row.find_all("td")))
datasets.append(dataset)
print datasets
The result looks like this:
结果如下所示：
[[(u'Tests', u'103'),
(u'Failures', u'24'),
(u'Success Rate', u'76.70%'),
(u'Average Time', u'71 ms'),
(u'Min Time', u'0 ms'),
(u'Max Time', u'829 ms')]]
Edit2:To produce the desired output, use something like this:
Edit2：要产生所需的输出，请使用以下内容：
for dataset in datasets:
for field in dataset:
print "{0:<16}: {1}".format(field[0], field[1])
Result:
结果：
Tests : 103
Failures : 24
Success Rate : 76.70%
Average Time : 71 ms
Min Time : 0 ms
Max Time : 829 ms
回答by cdtits
undef $/;
$text = <DATA>;
@tabs = $text =~ m!<table.*?>(.*?)</table>!gms;
for (@tabs) {
@th = m!<th>(.*?)</th>!gms;
@td = m!<td>(.*?)</td>!gms;
}
for $i (0..$#th) {
printf "%-16s\t: %s\n", $th[$i], $td[$i];
}
__DATA__
<table class="details" border="0" cellpadding="5" cellspacing="2" width="95%">
<tr valign="top">
<th>Tests</th>
<th>Failures</th>
<th>Success Rate</th>
<th>Average Time</th>
<th>Min Time</th>
<th>Max Time</th>
</tr>
<tr valign="top" class="Failure">
<td>103</td>
<td>24</td>
<td>76.70%</td>
<td>71 ms</td>
<td>0 ms</td>
<td>829 ms</td>
</tr>
</table>
output as follows:
输出如下：
Tests : 103
Failures : 24
Success Rate : 76.70%
Average Time : 71 ms
Min Time : 0 ms
Max Time : 829 ms
回答by mzjn
A Python solution that uses only the standard library (takes advantage of the fact that the HTML happens to be well-formed XML). More than one row of data can be handled.
仅使用标准库的 Python 解决方案（利用 HTML 恰好是格式良好的 XML 的事实）。可以处理多于一行的数据。
(Tested with Python 2.6 and 2.7. The question was updated saying that the OP uses Python 2.4, so this answer may not be very useful in this case. ElementTree was added in Python 2.5)
（用 Python 2.6 和 2.7 测试。问题已更新，说 OP 使用 Python 2.4，所以这个答案在这种情况下可能不是很有用。在 Python 2.5 中添加了 ElementTree）
from xml.etree.ElementTree import fromstring
HTML = """
<table class="details" border="0" cellpadding="5" cellspacing="2" width="95%">
<tr valign="top">
<th>Tests</th>
<th>Failures</th>
<th>Success Rate</th>
<th>Average Time</th>
<th>Min Time</th>
<th>Max Time</th>
</tr>
<tr valign="top" class="Failure">
<td>103</td>
<td>24</td>
<td>76.70%</td>
<td>71 ms</td>
<td>0 ms</td>
<td>829 ms</td>
</tr>
<tr valign="top" class="whatever">
<td>A</td>
<td>B</td>
<td>C</td>
<td>D</td>
<td>E</td>
<td>F</td>
</tr>
</table>"""
tree = fromstring(HTML)
rows = tree.findall("tr")
headrow = rows[0]
datarows = rows[1:]
for num, h in enumerate(headrow):
data = ", ".join([row[num].text for row in datarows])
print "{0:<16}: {1}".format(h.text, data)
Output:
输出：
Tests : 103, A
Failures : 24, B
Success Rate : 76.70%, C
Average Time : 71 ms, D
Min Time : 0 ms, E
Max Time : 829 ms, F
回答by Stephane Rouberol
Assuming your html code is stored in a mycode.html file, here is a bash way:
假设您的 html 代码存储在 mycode.html 文件中，这里是一种 bash 方式：
paste -d: <(grep '<th>' mycode.html | sed -e 's,</*th>,,g') <(grep '<td>' mycode.html | sed -e 's,</*td>,,g')
note: the output is not perfectly aligned
注意：输出没有完全对齐
回答by Michel Müller
Here is the top answer, adapted for Python3 compatibility, and improved by stripping whitespace in cells:
这是最佳答案，适用于 Python3 兼容性，并通过去除单元格中的空白进行改进：
from bs4 import BeautifulSoup
html = """
<table class="details" border="0" cellpadding="5" cellspacing="2" width="95%">
<tr valign="top">
<th>Tests</th>
<th>Failures</th>
<th>Success Rate</th>
<th>Average Time</th>
<th>Min Time</th>
<th>Max Time</th>
</tr>
<tr valign="top" class="Failure">
<td>103</td>
<td>24</td>
<td>76.70%</td>
<td>71 ms</td>
<td>0 ms</td>
<td>829 ms</td>
</tr>
</table>"""
soup = BeautifulSoup(s, 'html.parser')
table = soup.find("table")
# The first tr contains the field names.
headings = [th.get_text().strip() for th in table.find("tr").find_all("th")]
print(headings)
datasets = []
for row in table.find_all("tr")[1:]:
dataset = dict(zip(headings, (td.get_text() for td in row.find_all("td"))))
datasets.append(dataset)
print(datasets)
回答by paolov
Below is a python regex based solution that I have tested on python 2.7. It doesn't rely on xml module--so will work in case xml is not fully well formed.
下面是我在 python 2.7 上测试过的基于 python 正则表达式的解决方案。它不依赖于 xml 模块——因此可以在 xml 格式不完整的情况下工作。
import re
# input args: html string
# output: tables as a list, column max length
def extract_html_tables(html):
tables=[]
maxlen=0
rex1=r'<table.*?/table>'
rex2=r'<tr.*?/tr>'
rex3=r'<(td|th).*?/(td|th)>'
s = re.search(rex1,html,re.DOTALL)
while s:
t = s.group() # the table
s2 = re.search(rex2,t,re.DOTALL)
table = []
while s2:
r = s2.group() # the row
s3 = re.search(rex3,r,re.DOTALL)
row=[]
while s3:
d = s3.group() # the cell
#row.append(strip_tags(d).strip() )
row.append(d.strip() )
r = re.sub(rex3,'',r,1,re.DOTALL)
s3 = re.search(rex3,r,re.DOTALL)
table.append( row )
if maxlen<len(row):
maxlen = len(row)
t = re.sub(rex2,'',t,1,re.DOTALL)
s2 = re.search(rex2,t,re.DOTALL)
html = re.sub(rex1,'',html,1,re.DOTALL)
tables.append(table)
s = re.search(rex1,html,re.DOTALL)
return tables, maxlen
html = """
<table class="details" border="0" cellpadding="5" cellspacing="2" width="95%">
<tr valign="top">
<th>Tests</th>
<th>Failures</th>
<th>Success Rate</th>
<th>Average Time</th>
<th>Min Time</th>
<th>Max Time</th>
</tr>
<tr valign="top" class="Failure">
<td>103</td>
<td>24</td>
<td>76.70%</td>
<td>71 ms</td>
<td>0 ms</td>
<td>829 ms</td>
</tr>
</table>"""
print extract_html_tables(html)
回答by Jordan Valansi
Use pandas.read_html:
import pandas as pd
html_tables = pd.read_html('resources/test.html')
df = html_tables[0]
df.T # transpose to align
0
Tests 103
Failures 24
Success Rate 76.70%
Average Time 71 ms
